SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} A _ {i i} \leq \frac {5}{3} - \frac {1 0 (1 - 2 h) (1 + h) ^ {2} h ^ {2}}{2 h ^ {2} (3 + 8 h ^ {3}) + 9 h ^ {4}} \\ = \frac {5 h ^ {2} (2 7 + 2 8 h)}{3 (6 + 9 h ^ {2} + 1 6 h ^ {3})} \\ \leq \frac {5}{1 8} h ^ {2} (2 7 + 2 8 h) \\ = \frac {1 5}{2} \gamma^ {\frac {1}{2}} + \frac {7 0}{9} \gamma^ {\frac {3}{4}} \\ = \frac {1 5}{2} \sigma m ^ {- \frac {1}{2}} + \frac {7 0}{9} \sigma^ {\frac {3}{2}} m ^ {- \frac {3}{4}} \\ \end{array}
$$
where the third line drops  $h^2$  and  $h^3$  in the denominator; the forth line plugs in the value  $h = \gamma^{\frac{1}{4}}$  back and drops the constants. Summing over the coordinates  $1 \leq i \leq d$  gives the first inequality:
$$
E _ {d, k, \sigma} (2 m d) \lesssim \sigma d m ^ {- \frac {1}{2}} + \sigma^ {\frac {3}{2}} d m ^ {- \frac {3}{4}}
$$
For large enough  $m$ , the bound is dominated by the first term  $\sigma dm^{-\frac{1}{2}}$ , which completes the proof.
We end this section with a short summary. Lemma 4 and Lemma 5 happen to end up with the same rate  $E_{d,k,s}(2md) = \mathcal{O}(\sigma dm^{-\frac{1}{2}})$ . Replacing  $2md$  with the batch size  $b$ , we have shown that  $E_{d,k,s}(b) = \mathcal{O}\left(\sigma d^{\frac{3}{2}}b^{-\frac{1}{2}}\right)$  for the RBF kernel and  $\nu = 2.5$  Matérn kernel.
B.3 Discussion: Forward Differencing Designs
This section explores an alternative proof for the error function based on forward differencing designs, as opposed to the central differencing designs in Lemma 12. Similar to the previous section, we assume  $k(\mathbf{0},\mathbf{0}) = 1$ , which is indeed satisfied by the RBF kernel and Matérn kernel.
Lemma 13. Consider following  $(d + 1)m$  points  $\mathbf{Z} \in \mathbb{R}^{(d + 1)m \times d}$  defined as
$$
\mathbf {z} _ {j} ^ {(0)} = \mathbf {0}, \quad j = 1, 2, \dots , m
$$
$$
\mathbf {z} _ {j} ^ {(i)} = h \mathbf {e} _ {i}, \quad i \geq 1, \quad j = 1, 2, \dots , m
$$
where  $\mathbf{e}_i$  is the  $i$ -th standard unit vector. Define
$$
\operatorname {t r} \mathbf {A} \leq - \sum_ {i = 1} ^ {d} \left(\partial_ {i} ^ {2} \phi (\mathbf {0}) + \frac {(1 + \gamma) \beta_ {i} ^ {2}}{(1 + \gamma) ^ {2} - \alpha_ {i} ^ {2}}\right)
$$
where  $\alpha_{i} = \phi (h\mathbf{e}_{i})$ $\beta_{i} = \partial_{i}\phi (-h\mathbf{e}_{i})$  and  $\gamma = \frac{\sigma^2}{m}$
Proof. The proof is similar to Lemma 12.
Denote  $\mathbf{Z}^{(i)}$  be the subset of points consisting of  $\mathbf{z}_j^{(i)}$  and  $\mathbf{z}_j^{(0)}$ , where  $j = 1,2,\dots,m$ . Notice that the  $i$ -th diagonal entry  $A_{ii}$  can be bounded by
$$
A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \partial_ {i} k (\mathbf {0}, \mathbf {Z} ^ {(i)}) (k (\mathbf {Z} ^ {(i)}, \mathbf {Z} ^ {(i)}) + \sigma^ {2} \mathbf {I}) ^ {- 1} (\partial_ {i} k (\mathbf {0}, \mathbf {Z} ^ {(i)})) ^ {\top}.
$$
We need to invert the  $2m \times 2m$  matrix
$$
\begin{array}{l} \widehat {\mathbf {K}} = k (\mathbf {Z} ^ {(i)}, \mathbf {Z} ^ {(i)}) + \sigma^ {2} \mathbf {I} \\ = \left( \begin{array}{c c} \mathbf {1 1} ^ {\top} & \alpha_ {i} \mathbf {1} ^ {\top} \\ \alpha_ {i} \mathbf {1} & \mathbf {1 1} ^ {\top} \end{array} \right) + \sigma^ {2} \mathbf {I}. \\ \end{array}
$$
Again, we resort to the eigendecomposition of  $\widehat{\mathbf{K}} = \mathbf{Q}\mathbf{A}\mathbf{Q}^{\top}$ . The top two eigenvalues of  $\widehat{\mathbf{K}}$  are  $\lambda_{1} = m(1 + \alpha_{i}) + \sigma^{2}$  and  $\lambda_{2} = m(1 - \alpha_{i}) + \sigma^{2}$ . The remaining  $2m - 2$  eigenvalues are  $\sigma^{2}$ . The top two eigenvectors are
$$
\mathbf {q} _ {1} = \frac {1}{\sqrt {2 m}} \left( \begin{array}{c} \mathbf {1} \\ \mathbf {1} \end{array} \right), \quad \mathbf {q} _ {2} = \frac {1}{\sqrt {2 m}} \left( \begin{array}{c} \mathbf {1} \\ - \mathbf {1} \end{array} \right).
$$
Denote  $\mathbf{v}^{\top} = \partial_{i}k\big(\mathbf{0},\mathbf{Z}^{(i)}\big)$ . Note that  $\mathbf{v}$  can be written as a linear combination of  $\mathbf{q}_1$  and  $\mathbf{q}_2$ :
$$
\mathbf {v} = \beta_ {i} \left( \begin{array}{c} \mathbf {0} \\ \mathbf {1} \end{array} \right) = \frac {1}{2} \beta_ {i} \sqrt {2 m} (\mathbf {q} _ {1} - \mathbf {q} _ {2}).
$$
Then, straightforward calculation gives  $\mathbf{Q}^{\top}\mathbf{v} = \frac{1}{2}\beta_{i}\sqrt{2m}(1 - 10\cdots 0)^{\top}$ .
$$
\begin{array}{l} A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \mathbf {v} ^ {\top} \mathbf {Q} \boldsymbol {\Lambda} ^ {- 1} \mathbf {Q} ^ {\top} \mathbf {v} \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {1}{2} m \beta_ {i} ^ {2} \left(\frac {1}{m (1 + \alpha_ {i}) + \sigma^ {2}} + \frac {1}{m (1 - \alpha_ {i}) + \sigma^ {2}}\right) \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {1}{2} \beta_ {i} ^ {2} \left(\frac {1}{1 + \alpha_ {i} + \gamma} + \frac {1}{1 - \alpha_ {i} + \gamma}\right) \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {\beta_ {i} ^ {2} (1 + \gamma)}{(1 + \gamma) ^ {2} - \alpha_ {i} ^ {2}}. \\ \end{array}
$$
Summing over all coordinates finishes the proof.
Lemma 14. Let  $k(\mathbf{x}_1, \mathbf{x}_2) = \exp \left(-\frac{1}{2} \| \mathbf{x}_1 - \mathbf{x}_2 \|^2\right)$  be the RBF kernel. The forward differencing designs give a decay rate of  $E_{d,k,\sigma}(b) = \mathcal{O}(\sigma dm^{-\frac{1}{2}})$ .
Proof. Define  $\alpha = \phi(h\mathbf{e}_i)$  and  $\beta = \partial_i\phi(-h\mathbf{e}_i)$ . Plugging the values of  $\alpha$  and  $\beta$  into the bounds in Lemma 13 yields
$$
A _ {i i} \leq 1 - \frac {h ^ {2} \exp (- h ^ {2}) (1 + \gamma)}{(1 + \gamma) ^ {2} - \exp (- h ^ {2})}.
$$
The bound holds for arbitrary  $h$ . Applying the transformation  $h \mapsto \sqrt{h}$  gives
$$
\begin{array}{l} A _ {i i} \leq 1 - \frac {h \exp (- h) (1 + \gamma)}{(1 + \gamma) ^ {2} - \exp (- h)} \\ \leq 1 - \frac {h (1 - h) (1 + \gamma)}{(1 + \gamma) ^ {2} - (1 - h)} \\ = 1 - \frac {h (1 - h) (1 + \gamma)}{\gamma^ {2} + 2 \gamma + h} \\ \leq 1 - \frac {h (1 - h)}{\gamma^ {2} + 2 \gamma + h}, \\ \end{array}
$$
where the second line uses the inequality  $\exp(-h) \geq 1 - h$  in the numerator and the denominator; the last line is because  $\gamma$  is nonnegative. Let  $h = \gamma^{\frac{1}{2}}$  so that  $\gamma = h^2$ . Then we have
$$
\begin{array}{l} A _ {i i} \leq 1 - \frac {h (1 - h)}{h ^ {4} + 2 h ^ {2} + h} \\ = \frac {h ^ {3} + 3 h}{h ^ {3} + 2 h + 1} \\ \leq h ^ {3} + 3 h \\ = \gamma^ {\frac {3}{2}} + 3 \gamma^ {\frac {1}{2}} \\ \lesssim \gamma^ {\frac {1}{2}} \\ = \sigma m ^ {- \frac {1}{2}} \\ \end{array}
$$
where the first line plugs in  $\gamma = h^2$ ; the third line drops the  $h^3 + 2h$  in the denominator; the forth lines plugs in  $h = \gamma^{\frac{1}{2}}$ ; the fifth line is because  $\gamma^{\frac{1}{2}}$  dominates the bound when  $m$  is large. Thus, we have shown  $E_{d,k,\sigma}(dm + m) = \mathcal{O}(\sigma dm^{-\frac{1}{2}})$ .
The above lemma shows that forward differencing designs achieve the same asymptotic decay rate as the central differencing designs for the RBF kernel. Though, the leading constant in the big  $\mathcal{O}$  notation is slightly larger.
The following is a useful lemma for biased gradient updates, which bounds the gradient norm via the cumulative bias. The proof is adapted from a lemma by Ajalloeian and Stich [1, Lemma 2].
Lemma 15. Let  $f$  be  $L$ -smooth and bounded from below. Suppose the gradient oracle  $\hat{\mathbf{g}}_t$  has bias bounded by  $\xi_t$  in the  $t$ -th iteration. Namely, we have  $\| \hat{\mathbf{g}}_t - \mathbf{g}_t \|^2 \leq \xi_t$  for all  $t \geq 0$ , where  $\mathbf{g}_t = \nabla f(\mathbf{x}_t)$  is the ground truth gradient. Then the update  $\mathbf{x}_{t+1} = \mathbf{x}_t - \eta_t \hat{\mathbf{g}}_t$  with  $\eta_t \leq \frac{1}{L}$  produces a sequence  $\{\mathbf{x}_t\}_{t=1}^{\infty}$  satisfying
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {2 (f (\mathbf {x} _ {1}) - f ^ {*})}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {\sum_ {i = 1} ^ {T} \eta_ {t} \xi_ {t}}{\sum_ {t = 1} ^ {T} \eta_ {t}}
$$
Proof. By  $L$ -smoothness, we have
$$
f (\mathbf {x} _ {t + 1}) \leq f (\mathbf {x} _ {t}) + \nabla f (\mathbf {x} _ {t}) ^ {\top} (\mathbf {x} _ {t + 1} - \mathbf {x} _ {t}) + \frac {1}{2} L \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2}.
$$
Plugging in the update formula  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta_t\hat{\mathbf{g}}_t$  , we have
$$
\begin{array}{l} f \left(\mathbf {x} _ {t + 1}\right) \leq f \left(\mathbf {x} _ {t}\right) - \eta_ {t} \nabla f \left(\mathbf {x} _ {t}\right) ^ {\top} \hat {\mathbf {g}} _ {t} + \frac {1}{2} L \eta_ {t} ^ {2} \| \hat {\mathbf {g}} _ {t} \| ^ {2} \\ \leq f (\mathbf {x} _ {t}) - \eta_ {t} \nabla f (\mathbf {x} _ {t}) ^ {\top} \hat {\mathbf {g}} _ {t} + \frac {1}{2} \eta_ {t} \| \hat {\mathbf {g}} _ {t} \| ^ {2} \\ \leq f (\mathbf {x} _ {t}) - \eta_ {t} \nabla f (\mathbf {x} _ {t}) ^ {\top} \hat {\mathbf {g}} _ {t} + \frac {1}{2} \eta_ {t} \| \hat {\mathbf {g}} _ {t} - \nabla f (\mathbf {x} _ {t}) + \nabla f (\mathbf {x} _ {t}) \| ^ {2} \\ \leq f \left(\mathbf {x} _ {t}\right) + \frac {1}{2} \eta_ {t} \left(\| \hat {\mathbf {g}} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right) \| ^ {2} - \| \nabla f \left(\mathbf {x} _ {t}\right) \| ^ {2}\right) \\ \leq f (\mathbf {x} _ {t}) - \frac {1}{2} \eta_ {t} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} + \frac {1}{2} \eta_ {t} \xi_ {t}, \\ \end{array}
$$
where the first inequality uses  $L$ -smoothness; the second inequality uses  $\eta_t \leq \frac{1}{L}$ ; the fourth inequality expands the squared Euclidean norm; the last inequality uses the definition of bias. Summing the inequalities for  $t = 1, 2, \dots, T$  and rearranging the terms, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq 2 (f (\mathbf {x} _ {1}) - f (\mathbf {x} _ {T + 1})) + \sum_ {t = 1} ^ {T} \eta_ {t} \xi_ {t} \\ \leq 2 \left(f \left(\mathbf {x} _ {1}\right) - f ^ {*}\right) + \sum_ {t = 1} ^ {T} \eta_ {t} \xi_ {t}. \\ \end{array}
$$
Dividing both sides by  $\sum_{t=1}^{T} \eta_t$  gives
$$
\frac {\sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2}}{\sum_ {t = 1} ^ {T} \eta_ {t}} \leq \frac {2 (f (\mathbf {x} _ {1}) - f ^ {*})}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {\sum_ {t = 1} ^ {T} \eta_ {t} \xi_ {t}}{\sum_ {t = 1} ^ {T} \eta_ {t}}.
$$
The left hand side is a weighted average, which is greater than the minimum over  $1 \leq t \leq T$ , which completes the proof.
Next, we prove a variant of Lemma 15 for the projected update  $\mathbf{x}_{t + 1} = \mathrm{proj}_{\mathcal{X}}(\mathbf{x}_t - \eta_t\nabla f(\mathbf{x}_t))$ . For the ground truth gradient  $\mathbf{g}_t = \nabla f(\mathbf{x}_t)$ , define the gradient mapping
$$
G (\mathbf {x} _ {t}) = \frac {1}{\eta_ {t}} \big (\mathbf {x} _ {t} - \operatorname {p r o j} _ {\mathcal {X}} (\mathbf {x} _ {t} - \eta_ {t} \mathbf {g} _ {t}) \big).
$$
For the approximate gradient  $\hat{\mathbf{g}}_t = \nabla \mu_{\mathcal{D}_t}(\mathbf{x}_t)$ , define the gradient mapping
$$
\widehat {G} (\mathbf {x} _ {t}) = \frac {1}{\eta_ {t}} \bigl (\mathbf {x} _ {t} - \mathrm {p r o j} _ {\mathcal {X}} (\mathbf {x} _ {t} - \eta_ {t} \hat {\mathbf {g}} _ {t}) \bigr).
$$
In the following, we introduce two lemmas characterizing the projection operator  $\mathrm{proj}_{\mathcal{X}}(\cdot)$ .
Lemma 16 (e.g., Lemma 3.1 of Bubeck et al. [2]). Let  $\mathcal{X}$  be convex and compact. Let  $\mathbf{x} \in \mathcal{X}$  and  $\mathbf{z} \in \mathbb{R}^d$ . Then we have
$$
\left(\operatorname {p r o j} _ {\mathcal {X}} (\mathbf {z}) - \mathbf {z}\right) ^ {\top} \left(\mathbf {x} - \operatorname {p r o j} _ {\mathcal {X}} (\mathbf {z})\right) \geq 0.
$$
As a result,  $\| \mathbf{x} - \mathbf{z}\|^2 \geq \| \mathrm{proj}_{\mathcal{X}}(\mathbf{z}) - \mathbf{z}\|^2 + \| \mathbf{x} - \mathrm{proj}_{\mathcal{X}}(\mathbf{z})\|^2$ .
Lemma 17. The following holds:
1.  $\| G(\mathbf{x}_t)\| \leq \| \mathbf{g}_t\|$
2.  $\| G(\mathbf{x}_t) - \mathbf{g}\| \leq \| \mathbf{g}_t\|$
3.  $\| \widehat{G} (\mathbf{x}_t) - G(\mathbf{x}_t)\| \leq \| \hat{\mathbf{g}}_t - \mathbf{g}_t\|$
$$
4. \mathbf {g} ^ {\top} G (\mathbf {x} _ {t}) \geq \| G (\mathbf {x} _ {t}) \| ^ {2}.
$$
Proof. (1-2): The first two inequalities are direct corollaries of Lemma 16.
(3): This is proved as follows:
$$
\begin{array}{l} \| \widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t}) \| = \frac {1}{\eta_ {t}} \| \operatorname {p r o j} _ {\mathcal {X}} (\mathbf {x} _ {t} - \eta_ {t} \mathbf {g} _ {t}) - \operatorname {p r o j} _ {\mathcal {X}} (\mathbf {x} _ {t} - \eta_ {t} \hat {\mathbf {g}} _ {t}) \| \\ \leq \left\| \hat {\mathbf {g}} _ {t} - \mathbf {g} _ {t} \right\|, \\ \end{array}
$$
where the second line is because the projection operator is non-expansive.
(4): By Lemma 16, we have
$$
- \eta_ {t} G (\mathbf {x} _ {t}) ^ {\top} \left(\eta_ {t} G (\mathbf {x} _ {t}) - \eta_ {t} \mathbf {g} _ {t}\right) \geq 0.
$$
Rearranging the terms finishes the proof.
Now we give a lemma proving biased gradient update with the projection operator. The proof is adapted from a lemma by Shu et al. [22].
Lemma 18. Let  $f$  be  $L$ -smooth over a convex compact set  $\mathcal{X}$ . Moreover, assume the gradient norm  $\| \nabla f(\mathbf{x})\|$  is bounded by  $L^{\prime}$  on  $\mathcal{X}$ . Suppose the gradient oracle  $\hat{\mathbf{g}}_t$  has bias bounded by  $\xi_{t}$  in the  $t$ -th iteration:  $\| \hat{\mathbf{g}}_t - \mathbf{g}_t\|^2 \leq \xi_t$  for all  $t \geq 0$ , where  $\mathbf{g}_t = \nabla f(\mathbf{x}_t)$ . Then the update  $\mathbf{x}_{t+1} = \mathrm{proj}_{\mathcal{X}}(\mathbf{x}_t - \eta_t\hat{\mathbf{g}}_t)$  with  $\eta_t \leq \frac{1}{L}$  produces a sequence  $\{\mathbf{x}_t\}_{t=1}^\infty$  satisfying
$$
\min _ {1 \leq t \leq T} \| G (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {2 (f (\mathbf {x} _ {1}) - f ^ {*})}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {\sum_ {i = 1} ^ {T} \eta_ {t} \xi_ {t}}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {L ^ {\prime} \sum_ {i = 1} ^ {T} \eta_ {t} \sqrt {\xi_ {t}}}{\sum_ {t = 1} ^ {T} \eta_ {t}},
$$
where  $G(\mathbf{x}_t) = \frac{1}{\eta_t}\big(\mathbf{x}_t - \mathrm{proj}_{\mathcal{X}}(\mathbf{x}_t - \eta_t\mathbf{g}_t)\big)$  is the gradient mapping.
$$
\begin{array}{l} f \left(\mathbf {x} _ {t + 1}\right) - f \left(\mathbf {x} _ {t}\right) \leq \mathbf {g} _ {t} ^ {\top} \left(\mathbf {x} _ {t + 1} - \mathbf {x} _ {t}\right) + \frac {1}{2} L \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \\ = - \eta_ {t} \mathbf {g} _ {t} ^ {\top} \widehat {G} (\mathbf {x} _ {t}) + \frac {1}{2} L \eta_ {t} ^ {2} \| \widehat {G} (\mathbf {x} _ {t}) \| ^ {2} \\ \leq - \eta_ {t} \mathbf {g} _ {t} ^ {\top} \widehat {G} (\mathbf {x} _ {t}) + \frac {1}{2} \eta_ {t} \| \widehat {G} (\mathbf {x} _ {t}) \| ^ {2}, \\ \end{array}
$$
where the second line plugs in the update  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta_t\widehat{G} (\mathbf{x}_t)$  and the third line is due to  $\eta_t\leq \frac{1}{L}$ . Now we analyze the two terms separately. For the first term, we have
$$
\begin{array}{l} - \eta_ {t} \mathbf {g} _ {t} ^ {\top} \widehat {G} (\mathbf {x} _ {t}) = - \eta_ {t} \mathbf {g} _ {t} ^ {\top} \left(\widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t})\right) - \eta_ {t} \mathbf {g} _ {t} ^ {\top} G (\mathbf {x} _ {t}) \\ \leq - \eta_ {t} \mathbf {g} _ {t} ^ {\top} \left(\widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t})\right) - \eta_ {t} \left\| G (\mathbf {x} _ {t}) \right\| ^ {2}, \\ \end{array}
$$
where the second inequality uses Lemma 17. For the second term, we have
$$
\begin{array}{l} \frac {1}{2} \eta_ {t} \| \widehat {G} (\mathbf {x} _ {t}) \| ^ {2} = \frac {1}{2} \eta_ {t} \| \widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t}) + G (\mathbf {x} _ {t}) \| ^ {2} \\ = \frac {1}{2} \eta_ {t} \left\| \widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t}) \right\| ^ {2} + \eta_ {t} \left(\widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t})\right) ^ {\top} G (\mathbf {x} _ {t}) + \frac {1}{2} \eta_ {t} \left\| G (\mathbf {x} _ {t}) \right\| ^ {2} \\ \leq \frac {1}{2} \eta_ {t} \| \hat {\mathbf {g}} _ {t} - \mathbf {g} _ {t} \| ^ {2} + \eta_ {t} \left(\widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t})\right) ^ {\top} G (\mathbf {x} _ {t}) + \frac {1}{2} \eta_ {t} \| G (\mathbf {x} _ {t}) \| ^ {2} \\ \end{array}
$$
Summing them together, we have
$$
\begin{array}{l} f \left(\mathbf {x} _ {t + 1}\right) - f \left(\mathbf {x} _ {t}\right) \leq \frac {1}{2} \eta_ {t} \| \hat {\mathbf {g}} _ {t} - \mathbf {g} _ {t} \| ^ {2} + \eta_ {t} \left(\widehat {G} \left(\mathbf {x} _ {t}\right) - G \left(\mathbf {x} _ {t}\right)\right) ^ {\top} \left(G \left(\mathbf {x} _ {t}\right) - \mathbf {g} _ {t}\right) - \frac {1}{2} \eta_ {t} \| G \left(\mathbf {x} _ {t}\right) \| ^ {2} \\ \leq \frac {1}{2} \eta_ {t} \| \hat {\mathbf {g}} _ {t} - \mathbf {g} _ {t} \| ^ {2} + \eta_ {t} \left\| \widehat {G} (\mathbf {x} _ {t}) - G (\mathbf {x} _ {t}) \right\| \cdot \| \mathbf {g} _ {t} \| - \frac {1}{2} \eta_ {t} \left\| G (\mathbf {x} _ {t}) \right\| ^ {2} \\ \leq \frac {1}{2} \eta_ {t} \xi_ {t} + \eta_ {t} L ^ {\prime} \sqrt {\xi_ {t}} - \frac {1}{2} \eta_ {t} \| G (\mathbf {x} _ {t}) \| ^ {2}, \\ \end{array}
$$
where the second line uses Cauchy-Schwarz inequality.
A telescoping sum gives
$$
\sum_ {t = 1} ^ {T} \eta_ {t} \| G (\mathbf {x} _ {t}) \| ^ {2} \leq \sum_ {t = 1} ^ {T} \eta_ {t} \xi_ {t} + 2 L ^ {\prime} \sum_ {t = 1} ^ {T} \eta_ {t} \sqrt {\xi_ {t}} + 2 (f (\mathbf {x} _ {1}) - f (\mathbf {x} _ {T + 1})),
$$
$$
\min _ {1 \leq t \leq T} \| G (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {2 (f (\mathbf {x} _ {1}) - f ^ {*})}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {\sum_ {i = 1} ^ {T} \eta_ {t} \xi_ {t}}{\sum_ {t = 1} ^ {T} \eta_ {t}} + \frac {L ^ {\prime} \sum_ {i = 1} ^ {T} \eta_ {t} \sqrt {\xi_ {t}}}{\sum_ {t = 1} ^ {T} \eta_ {t}}.
$$
The rest of this section proves all theorems and their corollaries in the main paper.
Proof. By Lemma 1 and Assumption 3, we can bound the bias in the iteration  $t$  as
$$
\left\| \nabla f (\mathbf {x} _ {t}) - \nabla \mu_ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}) \right\| ^ {2} \leq B ^ {2} \mathrm {t r} \bigl (\nabla k _ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}, \mathbf {x} _ {t}) \nabla^ {\top} \bigr).
$$
By Lemma 8, the trace in the RHS can be bounded by the error function  $E_{d,k,\sigma}(b)$ . Thus, the gradient bias is  $B^{2}E_{k,k,\sigma}(b)$ . Applying Lemma 15 with  $\eta_t = \frac{1}{L}$  and  $\xi_t = B^2 E_{k,k,\sigma}(b)$  finishes the proof.
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {1}{T} \big (2 L (f (\mathbf {x} _ {1}) - f ^ {*}) \big).
$$
Proof. By Lemma 2, we have  $E_{d,k,\sigma}(d + 1) = 0$ . Plugging it into Theorem 1 gives the rate in the iteration number  $T$ . To get the rate in samples  $n$ , note that  $n = \sum_{t=1}^{T} (d + 1) = (d + 1)T$ . Plugging  $T = \frac{n}{d + 1}$  into the rate finishes the proof.
Proof. By Lemma 11, we have
$$
\left\| \nabla f (\mathbf {x} _ {t}) - \nabla \mu_ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}) \right\| ^ {2} \leq C _ {t} \operatorname {t r} \left(\nabla k _ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}, \mathbf {x} _ {t}) \nabla^ {\top}\right)
$$
with probability at least  $1 - \delta$ . The trace on the RHS can be further bounded by the error function  $E_{d,k,\sigma}(b_t)$  by Lemma 8. Applying the union bound, with probability at least  $1 - 2\delta$ , the inequality
$$
\| \nabla f (\mathbf {x} _ {t}) - \nabla \mu_ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}) \| ^ {2} \leq C _ {t} E _ {d, k, \sigma} (b _ {t})
$$
holds for all  $t \geq 0$  and  $f$  is  $L$ -smooth. Applying Lemma 15 with  $\eta_t = \frac{1}{L}$  and  $\xi = C_t E_{d,k,\sigma}(b_t)$  finishes the proof.
Proof. By Theorem 2, with probability at least  $1 - 2\delta$ , we have
$$
\min _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {1}{T} \big (2 L (f (\mathbf {x} _ {1}) - f ^ {*}) \big) + \frac {1}{T} \sum_ {t = 1} ^ {T} C _ {t} E _ {d, k, \sigma} (b _ {t})
$$
The proof boils down to bounding the average cumulative bias. The full details is in Appendix D.
$$
\text {i f} \quad b _ {t} = \left\{ \begin{array}{l l} d t; \\ d t ^ {2}, \end{array} \right. \quad \text {t h e n} \quad \min  _ {1 \leq t \leq T} \| G (\mathbf {x} _ {t}) \| ^ {2} = \left\{ \begin{array}{l l} \mathcal {O} \big (\sigma d ^ {\frac {5}{4}} n ^ {- \frac {1}{4}} \log n + \sigma^ {\frac {1}{2}} d ^ {\frac {5}{8}} n ^ {- \frac {1}{8}} \log n \big); \\ \mathcal {O} \big (\sigma d ^ {\frac {4}{3}} n ^ {- \frac {1}{3}} \log^ {2} n + \sigma^ {\frac {1}{2}} d ^ {\frac {2}{3}} n ^ {- \frac {1}{6}} \log n \big), \end{array} \right.
$$
Proof. Since  $f$  is twice differentiable on a compact set  $\mathcal{X}$ , its gradient norm  $\| \nabla f(\mathbf{x})\|$  attains a maximum. Thus, there exists a constant  $L'$  such that  $L' \geq \| \nabla f(\mathbf{x})\|$  for all  $\mathbf{x} \in \mathcal{X}$ . By Lemma 18 and a similar argument in Theorem 2, with probability at least  $1 - 2\delta$ , we have
$$
\min _ {1 \leq t \leq T} \| G (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {1}{T} \big (2 L (f (\mathbf {x} _ {1}) - f ^ {*}) \big) + \frac {1}{T} \sum_ {t = 1} ^ {T} C _ {t} ^ {(2)} E _ {d, k, \sigma} (b _ {t}) + \frac {1}{T} \sum_ {t = 1} ^ {T} C _ {t} ^ {(2)} \sqrt {E _ {d , k , \sigma} (b _ {t})},
$$
where  $C_t^{(1)}$  and  $C_t^{(2)}$  are constants growing in  $\mathcal{O}(\log t)$ . By Lemmas 4 and 5, plug in the error function  $E_{d,k,\sigma}(b) = \mathcal{O}(\sigma d^{\frac{3}{2}}b^{-\frac{1}{2}})$ . The rest of the proof follows a similar argument in Corollary 2, as shown in Appendix D.
Finally, we present a convergence result for GP sample path under noiseless assumption.
Theorem 4. For  $0 < \delta < 1$ , suppose  $f$  is a GP sample whose smoothness constant is  $L$  with probability at least  $1 - \delta$ . Assuming  $\sigma = 0$ , Algorithm 1 with batch size  $b_{t} = d + 1$  and step size  $\eta_{t} = \frac{1}{L}$  produces a sequence satisfying
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {2 L (f (\mathbf {x} _ {1}) - f ^ {*})}{T}
$$
with probability at least  $1 - 2\delta$
Proof. By Theorem 2, we have
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {2 L (f (\mathbf {x} _ {1}) - f ^ {*})}{T} + \frac {1}{T} \sum_ {t = 1} ^ {T} C _ {t} E _ {d, k, \sigma} (d + 1)
$$
with probability at least  $1 - 2\delta$ . By Lemma 2,  $E_{d,k,\sigma}(d + 1) = 0$ , the second term is essentially zero, which finishes the proof.
D Optimizing the Batch Size
In this section, we optimize the batch size in Theorem 2 and give explicit convergence rates. The discussion in this section will give a proof for Corollary 2.
For the RBF kernel and  $\nu = 2.5$  Matérn kernel, by Theorem 2, Lemma 4 and Lemma 5, we have shown the following bound
$$
\begin{array}{l} \min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \lesssim \frac {1}{T} + \frac {1}{T} \sum_ {t = 1} ^ {T} E _ {d, k, \sigma} (b _ {t}) \log t \\ \lesssim \frac {1}{T} + \frac {\sigma d ^ {\frac {3}{2}}}{T} \sum_ {t = 1} ^ {T} b _ {t} ^ {- \frac {1}{2}} \log t. \\ \end{array}
$$
We discuss polynomially growing batch size  $b_{t} = dt^{a}$ , where  $a > 0$ . Then we have
$$
\min _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \lesssim \frac {1}{T} + \sigma d \cdot \frac {1}{T} \sum_ {t = 1} ^ {T} t ^ {- \frac {1}{2} a} \log t.
$$
We discuss three cases:  $0 < a < 2$ ,  $a = 2$  and  $a > 2$ .
Case 1. When  $0 \leq a < 2$ , the infinite sum  $\sum_{t=1}^{T} t^{-\frac{1}{2}a} \log t$  diverges. Its growth speed is on the order of  $\mathcal{O}(T^{1 - \frac{1}{2}a} \log T)$ . The total number of samples  $n = \sum_{t=1}^{T} b_t = \mathcal{O}(dT^{a+1})$ , and thus  $T = \mathcal{O}(d^{-\frac{1}{a+1}} n^{\frac{1}{a+1}})$ . Thus, the rate is
$$
\begin{array}{l} \min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \lesssim T ^ {- 1} + \sigma d \cdot T ^ {- \frac {1}{2} a} \log T \\ \lesssim d ^ {\frac {1}{a + 1}} n ^ {- \frac {1}{a + 1}} + \sigma d ^ {\frac {3 a + 2}{2 (a + 1)}} n ^ {- \frac {a}{2 (a + 1)}} \log n \\ \lesssim \sigma d ^ {\frac {3 a + 2}{2 (a + 1)}} n ^ {- \frac {a}{2 (a + 1)}} \log n, \\ \end{array}
$$
where the last inequality uses the fact that the second term dominates the rate when  $0 < a < 2$ .