SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the edge case, where the minimizer of the acquisition function  $\mathrm{argmin}_{\mathbf{Z}}\alpha_{\mathrm{trace}}(\mathbf{0},\mathbf{Z})$  does not exist (e.g., when  $\sigma = 0$ ), can be handled by a careful limiting argument using the same idea.
A.1 Lemmas for the RKHS Assumption
By Assumption 1, the kernel function  $k$  is four times continuously differentiable. The following lemma asserts the smoothness of  $f \in \mathcal{H}$ .
Lemma 9. Suppose  $f \in \mathcal{H}$  maps from a compact domain  $\mathcal{X}$  to  $\mathbb{R}$ . Then  $f$  is  $L$ -smooth for some  $L$ .
Proof. Since the kernel  $k$  is four times continuously differentiable,  $f$  is twice continuously differentiable. On a compact domain  $\mathcal{X}$ , the spectral norm of the Hessian  $\| \nabla f(\mathbf{x})\|$  has a maximizer. Define  $L = \max_{\mathbf{x}\in \mathcal{X}}\| \nabla f(\mathbf{x})\|$ . Then  $f$  is  $L$ -smooth.
Lemma 1. For any  $f\in \mathcal{H}$ , any  $\mathbf{x}\in \mathcal{X}$  and any  $\mathcal{D}$ , we have the following inequality
Proof. This is a simple corollary of a standard result in meshless scattered data approximation [e.g., 28, Theorem 11.4]. The main idea is to express the estimation error as a linear functional, and then compute the operator norm of that linear functional.
Let  $\lambda = \delta_{\mathbf{x}}D:\mathcal{H}\to \mathbb{R}$  be the composition of the evaluation operator and differential operator, i.e.  $\lambda f = Df(\mathbf{x})$ . Wendland [28, Theorem 11.4] provides a bound on  $(\lambda f - \lambda \mu_{\mathcal{D}})^{2}$ :
$$
\left(\lambda f - \lambda \mu_ {\mathcal {D}}\right) ^ {2} \leq \lambda^ {(1)} \lambda^ {(2)} k _ {\mathcal {D}} (\cdot , \cdot) \| f \| _ {\mathcal {H}} ^ {2},
$$
where  $\lambda^{(1)}$  applies  $\lambda$  to the first argument of  $k_{\mathcal{D}}$  and  $\lambda^{(2)}$  applies  $\lambda$  to the second argument of  $k_{\mathcal{D}}$ .
Pick the linear functional  $\lambda : f \mapsto \frac{\partial}{\partial x_i} f(\mathbf{x})$  where  $1 \leq i \leq d$ . Then, the left hand side becomes  $(\lambda f - \lambda \mu_{\mathcal{D}})^2 = (\frac{\partial}{\partial x_i} f(\mathbf{x}) - \frac{\partial}{\partial x_i} \mu_{\mathcal{D}}(\mathbf{x}))^2$ . The right hand side  $\lambda^{(1)} \lambda^{(2)} k_{\mathcal{D}}(\cdot, \cdot)$  is exactly the  $i$ -th diagonal entry of  $\nabla k_{\mathcal{D}}(\mathbf{x}, \mathbf{x}) \nabla^\top$ . For each  $i$ , use the above inequality, and then summing over all coordinates finishes the proof.
A.2 Lemmas for Convergence on GP Sample Paths
In this section, we provide a few lemmas for the GP sample path  $f \sim \mathcal{GP}(0,k)$ . By Assumption 1, we have the following lemma which asserts that  $f$  is smooth with high probability.
Proof. The proof uses the Borell-TIS inequality. Let  $f \sim \mathcal{GP}(0, k)$  be a Gaussian process. Provided that  $\sup_{\mathbf{x} \in \mathcal{X}} |f(\mathbf{x})|$  is almost surely finite, the Borell-TIS inequality states that
$$
\operatorname * {P r} (\sup  _ {\mathbf {x} \in \mathcal {X}} | f (\mathbf {x}) | > u + \mathbb {E} \sup  _ {\mathbf {x} \in \mathcal {X}} | f (\mathbf {x}) |) \leq \exp (- \frac {u ^ {2}}{2 s ^ {2}}),
$$
where  $u > 0$  is an arbitrary positive constant and  $s^2 = \sup_{\mathbf{x} \in \mathcal{X}} \mathbb{E}|f(\mathbf{x})|^2$ . Namely, if the supremum of  $f$  is almost surely finite then the supremum of  $f$  is bounded with high probability.
Since  $k$  is four time continuously differentiable, the second-order derivative  $\frac{\partial^2}{\partial x_i \partial x_j} f$  exists and is almost surely continuous. On a compact domain  $\mathcal{X}$ , the supremum  $\sup_{\mathbf{x} \in \mathcal{X}} |\frac{\partial^2}{\partial x_i \partial x_j} f(\mathbf{x})|$  is almost surely finite. By the Borell-TIS inequality, the expectation  $\mathbb{E} \sup_{\mathbf{x} \in \mathcal{X}} |\frac{\partial^2}{\partial x_i \partial x_j} f(\mathbf{x})|$  is finite and the supremum  $\sup_{\mathbf{x} \in \mathcal{X}} |\frac{\partial^2}{\partial x_i \partial x_j} f(\mathbf{x})|$  is bounded with high probability. Thus, the Frobenius norm of the Hessian  $\nabla^2 f(\mathbf{x})$  is bounded with high probability by a union bound. Since the spectral norm of  $\nabla f(\mathbf{x})$  can be bounded by its Frobenius norm, the spectral norm of the Hessian  $\| \nabla f(\mathbf{x}) \|$  is also bounded with high probability, which gives the smoothness constant.
The next two lemmas bound the gradient estimation error with high probability.
Lemma 10. Let  $\mathbf{u} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma})$  be a Gaussian vector. Then for any  $t > 0$
$$
\Pr (\| \mathbf {u} \| > t) \leq 2 \exp \left(- \frac {t ^ {2}}{2 \operatorname {t r} \boldsymbol {\Sigma}}\right).
$$
Proof. This is a standard concentration inequality result, but we give a self-contained proof here for completeness. Denote the spectral decomposition  $\boldsymbol{\Sigma} = \mathbf{Q}\boldsymbol{\Lambda}\mathbf{Q}^{\top}$ . By Markov's inequality, we have
$$
\begin{array}{l} \Pr (\| \mathbf {u} \| > t) = \Pr (e ^ {s \| \mathbf {u} \|} > e ^ {s t}) \\ \leq e ^ {- s t} \mathbb {E} e ^ {s \| \mathbf {u} \|}, \\ \end{array}
$$
where  $s > 0$  is an arbitrary positive constant. Let  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  be a standard Gaussian variable. Then it is easy to see that  $\| \mathbf{u}\| = \| \boldsymbol{\Lambda}^{\frac{1}{2}}\boldsymbol {\epsilon}\|$ . Then, replacing  $\| \mathbf{u}\|$  with  $\| \boldsymbol{\Lambda}^{\frac{1}{2}}\boldsymbol {\epsilon}\|$  gives
$$
\begin{array}{l} \Pr (\| \mathbf {u} \| > t) \leq e ^ {- s t} \mathbb {E} e ^ {s \| \boldsymbol {\Lambda} ^ {\frac {1}{2}} \boldsymbol {\epsilon} \|} \\ \leq e ^ {- s t} \mathbb {E} e ^ {s \| \boldsymbol {\Lambda} ^ {\frac {1}{2}} \boldsymbol {\epsilon} \| _ {1}} \\ = e ^ {- s t} \prod_ {i = 1} ^ {d} \mathbb {E} e ^ {s \sqrt {\lambda_ {i}} | \epsilon_ {i} |} \\ \leq 2 e ^ {- s t} \prod_ {i = 1} ^ {d} \mathbb {E} e ^ {s \sqrt {\lambda_ {i}} \epsilon_ {i}} \\ = 2 e ^ {- s t + \frac {1}{2} s ^ {2} \sum_ {i = 1} ^ {d} \lambda_ {i}}, \\ \end{array}
$$
where the first line plugs in  $\epsilon$ ; the second line uses  $\| \cdot \|_2 \leq \| \cdot \|_1$ ; the third line is due to independence of  $\epsilon_i$ ; the forth line removes the absolute value resulting an extra factor of 2; the last lines use the moment generating function of  $\epsilon_i$ . Optimizing the bound over  $s$  gives the desired result
$$
\operatorname * {P r} (\| \mathbf {u} \| > t) \leq 2 e ^ {- \frac {t ^ {2}}{2 \sum_ {i = 1} ^ {d} \lambda_ {i}}} = 2 \exp (- \frac {t ^ {2}}{2 \operatorname {t r} \boldsymbol {\Sigma}}).
$$
Lemma 11. For any  $0 < \delta < 1$ , let  $C_t = 2\log \left(\frac{\pi^2t^2}{6\delta}\right)$ . Then, the inequalities
$$
\left\| \nabla f (\mathbf {x} _ {t}) - \nabla \mu_ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}) \right\| ^ {2} \leq C _ {t} \operatorname {t r} \nabla k _ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}, \mathbf {x} _ {t}) \nabla^ {\top}
$$
hold for any  $t \geq 1$  with probability at least  $1 - \delta$ .
Proof. Since  $\nabla f(\mathbf{x}_t) \sim \mathcal{N}(\nabla \mu_{\mathcal{D}_t}(\mathbf{x}_t), \nabla k_{\mathcal{D}_t}(\mathbf{x}_t, \mathbf{x}_t)\nabla^\top)$ , applying Lemma 10 gives
$$
\Pr (\| \nabla f (\mathbf {x} _ {t}) - \nabla \mu_ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}) \| ^ {2} \geq C _ {t} \operatorname {t r} (\nabla k _ {\mathcal {D} _ {t}} (\mathbf {x} _ {t}, \mathbf {x} _ {t}) \nabla^ {\top})) \leq 2 \exp (- \frac {1}{2} C _ {t}).
$$
The particular choice of  $C_t$  makes the probability on the right hand side become  $\frac{6\delta}{\pi^2t^2}$ . Using the union bound over all  $t \geq 1$  and using the infinite sum  $\sum_{t=1}^{\infty} \frac{1}{t^2} = \frac{\pi^2}{6}$  finishes the proof.
We provide an important remark. The probability in Lemma 11 is taken over the randomness of  $f$  and the observation noise. On the other hand, the posterior mean gradient  $\nabla \mu_{\mathcal{D}_t}$  is deterministic, since it is conditioned on the data  $\mathcal{D}_t$ .
B Bounds on the Error Function  $E_{d,k,\sigma}$
This section is devoted to bounding the error function  $E_{d,k,\sigma}(b)$  in terms of the batch size  $b$ . The results in this section immediately give a bound on the posterior covariance trace by Lemma 8.
Before diving into the proofs, we present some immediate corollaries of Assumption 1 on the kernel. Because  $k$  is stationary, the kernel can be written as  $k(\mathbf{x},\mathbf{x}^{\prime}) = \phi (\mathbf{x} - \mathbf{x}^{\prime})$  for some positive-definite function  $\phi$ . Observe that  $\nabla k(\mathbf{x},\mathbf{x}^{\prime}) = \nabla \phi (\mathbf{x} - \mathbf{x}^{\prime})$  and  $\nabla k(\mathbf{x},\mathbf{x}^{\prime})\nabla^{\top} = -\nabla^{2}\phi (\mathbf{x} - \mathbf{x}^{\prime})$ . Denote the first-order partial derivative  $\partial_i\phi (\mathbf{x}) = \frac{\partial}{\partial x_i}\phi (\mathbf{x})$  and the second-order partial derivative  $\partial_i^2\phi (\mathbf{x}) = \frac{\partial^2}{\partial x_i^2}\phi (\mathbf{x})$ . It is easy to see that  $\phi$  is an even function and  $\nabla \phi$  is an odd function. In addition,  $\mathbf{0}$  is a maximum of  $\phi$ . Therefore,  $\nabla \phi (\mathbf{0}) = \mathbf{0}$  and the Hessian  $\nabla^2\phi (\mathbf{0})$  is negative semi-definite.
B.1 Noiseless Setting
The following is a bound for the error function  $E_{d,k,0}$  for arbitrary kernels satisfying Assumption 1 in the noiseless setting  $\sigma = 0$ .
Proof. The bound holds trivially for  $b = 0, 1$  and thus a proof is only needed for  $b \geq 2$ , which we split into two cases  $2 \leq b \leq d + 1$  and  $b > d + 1$ .
We first focus on the case  $2 \leq b \leq d + 1$ . Let  $\mathbf{z}_0 = \mathbf{0}$  and  $\mathbf{z}_i = h\mathbf{e}_i$  where  $i = 1, 2, \dots, b - 1$ , where  $\mathbf{e}_i$  is the  $i$ -th standard unit vector and  $h > 0$  is a constant. Define  $\mathbf{Z} = (\mathbf{z}_0 \quad \mathbf{z}_1 \quad \dots \quad \mathbf{z}_{b-1})^\top$ . By the definition of the error function, we have
$$
\begin{array}{l} E _ {d, k, 0} (b) \leq \operatorname {t r} \left(\nabla k (\mathbf {0}, \mathbf {0}) \nabla^ {\top} - \nabla k (\mathbf {0}, \mathbf {Z}) k (\mathbf {Z}, \mathbf {Z}) ^ {- 1} k (\mathbf {Z}, \mathbf {0}) \nabla^ {\top}\right) \\ = \sum_ {i = 1} ^ {d} A _ {i i} \\ = C (1 + d - b) + \sum_ {i = 1} ^ {b - 1} A _ {i i}, \\ \end{array}
$$
where we define  $\mathbf{A} = \nabla k(\mathbf{0},\mathbf{0})\nabla^{\top} - \nabla k(\mathbf{0},\mathbf{Z})k(\mathbf{Z},\mathbf{Z})^{-1}k(\mathbf{Z},\mathbf{0})\nabla^{\top}$  and use the inequality  $A_{ii}\leq -\partial_i^2\phi (\mathbf{0})\leq C$  for  $b\leq i\leq d$
Let us focus on the  $i$ -th diagonal entry  $A_{ii}$  where  $1 \leq i \leq b - 1$ . Then we have
$$
\begin{array}{l} A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - (0 \quad \partial_ {i} \phi (- h \mathbf {e} _ {i})) \left( \begin{array}{c c} \phi (\mathbf {0}) & \phi (h \mathbf {e} _ {i}) \\ \phi (h \mathbf {e} _ {i}) & \phi (\mathbf {0}) \end{array} \right) ^ {- 1} \left( \begin{array}{c} 0 \\ \partial_ {i} \phi (- h \mathbf {e} _ {i}) \end{array} \right) \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {1}{(\phi (\mathbf {0})) ^ {2} - (\phi (h \mathbf {e} _ {i})) ^ {2}} (0 \quad \partial_ {i} \phi (- h \mathbf {e} _ {i})) \left( \begin{array}{c c} \phi (\mathbf {0}) & - \phi (h \mathbf {e} _ {i}) \\ - \phi (h \mathbf {e} _ {i}) & \phi (\mathbf {0}) \end{array} \right) \left( \begin{array}{c} 0 \\ \partial_ {i} \phi (- h \mathbf {e} _ {i}) \end{array} \right) \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {\phi (\mathbf {0}) \left(\partial_ {i} \phi \left(h \mathbf {e} _ {i}\right)\right) ^ {2}}{\left(\phi (\mathbf {0})\right) ^ {2} - \left(\phi \left(h \mathbf {e} _ {i}\right)\right) ^ {2}}, \\ \end{array}
$$
where the first line is because conditioning on the subset  $\mathbf{z}_0$  and  $\mathbf{z}_i$  does not make the posterior smaller. Now let  $h\to 0$  and compute the limit by L'Hôpital's rule. We have
$$
\begin{array}{l} \lim  _ {h \rightarrow 0} A _ {i i} (h) = \lim  _ {h \rightarrow 0} - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {\phi (\mathbf {0}) \left(\partial_ {i} \phi \left(h \mathbf {e} _ {i}\right)\right) ^ {2}}{\left(\phi (\mathbf {0})\right) ^ {2} - \left(\phi \left(h \mathbf {e} _ {i}\right)\right) ^ {2}} \\ = \lim  _ {h \rightarrow 0} - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {\phi (\mathbf {0})}{\phi (\mathbf {0}) + \phi (h \mathbf {e} _ {i})} \cdot \frac {2 \partial_ {i} \phi (h \mathbf {e} _ {i}) \partial_ {i} ^ {2} \phi (h \mathbf {e} _ {i})}{- \partial_ {i} \phi (h \mathbf {e} _ {i})} \\ = 0. \\ \end{array}
$$
Thus letting  $h \to 0$  gives the inequality  $E_{d,k,\sigma}(b) \leq C(1 + d - b)$  for  $2 \leq b \leq d + 1$ .
When  $d > d + 1$ , note that  $E_{d,k,\sigma}(b)$  is an decreasing function in  $b$  and thus  $E_{d,k,\sigma}(b) \leq E_{d,k,\sigma}(d + 1) = 0$ . Both cases can be bounded by the expression  $C \max \{0, 1 + d - b\}$ .
B.2 Noisy Setting
This section proves bounds on the error function  $E_{d,k,\sigma}$  for the RBF kernel and the  $\nu = \frac{5}{2}$  Matérn kernel in the noisy setting. The lemmas in this section will implicitly use the assumption that  $k(\mathbf{0},\mathbf{0}) = 1$ . This assumption is indeed satisfied by the RBF kernel and the Matérn kernel, which are of primary concern in this paper.
Before proving the bound on  $E_{d,k,\sigma}$ , we need one more technical lemma:
Lemma 12 (Central Differencing Designs). Consider the  $2md$  points  $\mathbf{Z} \in \mathbb{R}^{2md \times d}$  defined as
$$
\mathbf {z} _ {j} ^ {(i)} = \left\{ \begin{array}{c l} - h \mathbf {e} _ {i}, & j = 1, 2, \dots m \\ h \mathbf {e} _ {i}, & j = m + 1, m + 2 \dots 2 m, \end{array} \right.
$$
where  $1 \leq i \leq d, 1 \leq j \leq 2m$  and  $\mathbf{e}_i$  is the  $i$ -th standard unit vector. Define
$$
\mathbf {A} = \nabla k (\mathbf {0}, \mathbf {0}) \nabla^ {\top} - \nabla k (\mathbf {0}, \mathbf {Z}) (k (\mathbf {Z}, \mathbf {Z}) + \sigma^ {2} \mathbf {I}) ^ {- 1} k (\mathbf {Z}, \mathbf {0}) \nabla^ {\top}.
$$
$$
A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {2 \beta_ {i} ^ {2}}{(1 - \alpha_ {i}) + \gamma},
$$
for all  $1 \leq i \leq d$  and thus
$$
\operatorname {t r} (\mathbf {A}) \leq - \sum_ {i = 1} ^ {d} \left(\partial_ {i} ^ {2} \phi (\mathbf {0}) + \frac {2 \beta_ {i} ^ {2}}{(1 - \alpha_ {i}) + \gamma}\right),
$$
where  $\alpha_{i} = \phi (2h\mathbf{e}_{i})$ $\beta_{i} = \partial_{i}\phi (-h\mathbf{e}_{i})$  and  $\gamma = \frac{\sigma^2}{m}$
Proof. Note that  $\mathbf{A}$  is the posterior covariance at the origin  $\mathbf{0}$  conditioned on  $\mathbf{Z}$ . Denote  $\mathbf{Z}^{(i)} = \left(\mathbf{z}_1^{(i)}\mathbf{z}_2^{(i)}\dots \mathbf{z}_m^{(i)}\mathbf{z}_{m + 1}^{(i)}\dots \mathbf{z}_{2m}^{(i)}\right)^\top$  the subset of  $2m$  points that lie on the  $i$ -th axis. Then, the  $i$ -th diagonal entry  $A_{ii}$  can be bounded by
$$
A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \partial_ {i} k (\mathbf {0}, \mathbf {Z} ^ {(i)}) (k (\mathbf {Z} ^ {(i)}, \mathbf {Z} ^ {(i)}) + \sigma^ {2} \mathbf {I}) ^ {- 1} (\partial_ {i} k (\mathbf {0}, \mathbf {Z} ^ {(i)})) ^ {\top},
$$
since conditioning on only a subset of points  $\mathbf{Z}^{(i)}$  would not make the posterior variances smaller (e.g., the posterior covariance is the Schur complement of a positive definite matrix). The remaining proof is dedicated to bounding the right hand side.
First, we need to compute the inverse of the  $2m \times 2m$  kernel matrix
$$
\begin{array}{l} \widehat {\mathbf {K}} = k \left(\mathbf {Z} ^ {(i)}, \mathbf {Z} ^ {(i)}\right) + \sigma^ {2} \mathbf {I} \\ = \left( \begin{array}{c c} \mathbf {1 1} ^ {\top} & \alpha_ {i} \mathbf {1 1} ^ {\top} \\ \alpha_ {i} \mathbf {1 1} ^ {\top} & \mathbf {1 1} ^ {\top} \end{array} \right) + \sigma^ {2} \mathbf {I}, \\ \end{array}
$$
where  $\alpha_{i} = \phi (2h\mathbf{e}_{i})$  is a nonnegative constant and  $\mathbf{1}$  is a  $m$  dimensional vector (we drop the index  $i$  in the matrix  $\widehat{\mathbf{K}}$  for notation simplicity). We compute the inverse analytically by forming its eigendecomposition
$$
\widehat {\mathbf {K}} = \mathbf {Q} \boldsymbol {\Lambda} \mathbf {Q} ^ {\top},
$$
where  $\pmb{\Lambda} = \mathrm{diag}(\lambda_1, \lambda_2, \dots, \lambda_{2m})$  and  $\mathbf{Q} = (\mathbf{q}_1 \quad \mathbf{q}_2 \quad \dots \quad \mathbf{q}_{2m})$ . Observe that:
$$
\widehat {\mathbf {K}} = \left( \begin{array}{c c} 1 & \alpha_ {i} \\ \alpha_ {i} & 1 \end{array} \right) \otimes \mathbf {1 1} ^ {\top} + \sigma^ {2} \mathbf {I},
$$
where  $\otimes$  denotes the Kronecker product. Because the eigenvalues (vectors) of a Kronecker product equal the Kronecker product of the individual eigenvalues (vectors), and because adding a diagonal shift simply shifts the eigenvalues, the top two eigenvalues of  $\widehat{\mathbf{K}}$  are  $\lambda_{1} = m(1 + \alpha_{i}) + \sigma^{2}$  and  $\lambda_{2} = m(1 - \alpha_{i}) + \sigma^{2}$ . The remaining  $2m - 2$  eigenvalues are  $\sigma^2$ . The top two eigenvectors are
$$
\mathbf {q} _ {1} = \frac {1}{\sqrt {2 m}} \left( \begin{array}{c} \mathbf {1} \\ \mathbf {1} \end{array} \right), \quad \mathbf {q} _ {2} = \frac {1}{\sqrt {2 m}} \left( \begin{array}{c} \mathbf {1} \\ - \mathbf {1} \end{array} \right),
$$
Next, we cope with the term  $\partial_i k(\mathbf{0},\mathbf{Z}^{(i)})$  , where the partial derivative is taken w.r.t. the first argument's  $i$  -th coordinate. Denote  $\mathbf{v}^{\top} = \partial_{i}k(\mathbf{0},\mathbf{Z}^{(i)})$  . Then it is easy to see that
$$
\mathbf {v} = \beta_ {i} \left( \begin{array}{c} - \mathbf {1} \\ \mathbf {1} \end{array} \right),
$$
where  $\beta_{i} = \partial_{i}\phi (-h\mathbf{e}_{i})$ . Note that  $\mathbf{v}$  happens to be an eigenvector of  $\widehat{\mathbf{K}}$  as well, because  $\mathbf{v} \parallel \mathbf{q}_2$ . As a result,  $\mathbf{Q}^{\top}\mathbf{v}$  has a simple expression  $\mathbf{Q}^{\top}\mathbf{v} = (0 - \sqrt{2m}\beta_{i} 0 \dots 0)^{\top}$ . Thus, we have
$$
\begin{array}{l} A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \mathbf {v} ^ {\top} \mathbf {Q} \boldsymbol {\Lambda} ^ {- 1} \mathbf {Q} ^ {\top} \mathbf {v} \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {2 m \beta_ {i} ^ {2}}{m (1 - \alpha_ {i}) + \sigma^ {2}} \\ = - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {2 \beta_ {i} ^ {2}}{(1 - \alpha_ {i}) + \gamma}. \\ \end{array}
$$
Summing over the coordinates  $1 \leq i \leq d$  finishes the proof.
With Lemma 12, we are finally ready to present the bounds for the RBF kernel and the Matérn kernel.
Proof. For any  $\mathbf{Z} \in \mathbb{R}^{2md \times d}$ , the following inequality holds by the definition of error function
$$
E _ {d, k, \sigma} (b) \leq \operatorname {t r} \left(\nabla k (\mathbf {0}, \mathbf {0}) \nabla^ {\top} - \nabla k (\mathbf {0}, \mathbf {Z}) \left(k (\mathbf {Z}, \mathbf {Z}) + \sigma^ {2} \mathbf {I}\right) ^ {- 1} k (\mathbf {Z}, \mathbf {0}) \nabla^ {\top}\right).
$$
Consider the following points  $\mathbf{z}_j^{(i)}$  defined as
where  $1 \leq i \leq d$ . The total number of points is exactly  $2md$ . By Lemma 12, we have
$$
E _ {d, k, \sigma} (b) \leq - \sum_ {i = 1} ^ {d} \left(\partial_ {i} ^ {2} \phi (\mathbf {0}) + \frac {2 \beta_ {i} ^ {2}}{(1 - \alpha_ {i}) + \gamma}\right).
$$
For the RBF kernel, the values of  $\alpha_{i}$  and  $\beta_{i}$  are the same for each coordinate  $1\leq i\leq d$  since it is isotropic:
$$
\alpha = \alpha_ {i} = \phi (2 h \mathbf {e} _ {i}) = \exp (- 2 h ^ {2}), \quad \beta = \beta_ {i} = \partial_ {i} \phi (- h \mathbf {e} _ {i}) = \exp \Big (- \frac {1}{2} h ^ {2} \Big) h.
$$
Plugging the value of  $\alpha, \beta, \gamma$  and  $\partial_i^2\phi(\mathbf{0})$  into the bound on  $E_{d,k,\sigma}$ , we have
$$
\begin{array}{l} A _ {i i} \leq 1 - \frac {2 m \exp (- h ^ {2}) h ^ {2}}{m (1 - \exp (- 2 h ^ {2})) + \sigma^ {2}} \\ \leq 1 - \frac {2 m \exp (- 2 h ^ {2}) h ^ {2}}{m (1 - \exp (- 2 h ^ {2})) + \sigma^ {2}} \\ \end{array}
$$
where the second inequality replaces  $\exp(-h^2)$  with  $\exp(-2h^2)$  in the numerator. Because the bound holds for arbitrary  $h$ , we can apply the transformation  $h \mapsto \frac{1}{\sqrt{2}} h$ , which gives the inequality
$$
A _ {i i} \leq 1 - \frac {m \exp (- h ^ {2}) h ^ {2}}{m (1 - \exp (- h ^ {2})) + \sigma^ {2}}.
$$
Our goal is to bound  $A_{ii}$  in terms of  $m$  and  $\sigma^2$ . Therefore, we minimize the right hand side over  $h$ . Define  $g(x) = 1 - \frac{me^{-x}x}{m(1 - e^{-x}) + \sigma^2}$ , where  $x \geq 0$ . The derivative is given by:
$$
g ^ {\prime} (x) = \frac {m (m + (m + \sigma^ {2}) e ^ {x} (x - 1))}{(m (e ^ {x} - 1) + \sigma^ {2} e ^ {x}) ^ {2}}.
$$
The unique stationary point is  $x^{*} = 1 + W\left(-\frac{m}{e(m + \sigma^{2})}\right)$ , where  $W$  is the principal branch of the Lambert  $W$  function. It is easy to see the stationary point  $x^{*}$  is the global minimizer of  $g(x)$  over  $\mathbb{R}_{\geq 0}$ . Plug  $x^{*}$  into the expression of  $g$ . Coincidentally, we have  $g(x^{*}) = 1 + W\left(-\frac{m}{e(m + \sigma^{2})}\right)$  as well  $-x^{*}$  is a fixed point of  $g$ .
In summary, we have shown  $A_{ii} \leq 1 + W\left(-\frac{m}{e(m + \sigma^2)}\right)$  for each coordinate  $i$ . Summing  $A_{ii}$  over all  $d$  coordinates proves the first inequality  $E_{d,k,\sigma}(2md) \leq d\left(1 + W\left(-\frac{m}{e(m + \sigma^2)}\right)\right)$ . The second inequality is a direction implication of Lemma 7, which completes the proof.
Proof. The proof is similar to Lemma 4. The difference is that we need to upper bound  $\partial_i^2\phi (\mathbf{0}) + \frac{2\beta_i^2}{(1 - \alpha_i) + \gamma}$  by a rational function. Otherwise the expression is intractable to minimize.
The Matérn kernel with half integer  $\nu$  can be written as a product of an exponential and a polynomial. In particular, for  $\nu = \frac{5}{2}$ , we have
$$
\phi (\mathbf {x}) = (1 + \sqrt {5} \| \mathbf {x} \| + \frac {5}{3} \| \mathbf {x} \| ^ {2}) \exp (- \sqrt {5} \| \mathbf {x} \)).
$$
When  $\mathbf{x}$  is in the nonnegative orthant, the gradient is
$$
\begin{array}{l} \nabla \phi (\mathbf {x}) = \exp (- \sqrt {5} \| \mathbf {x} \|) (\frac {\sqrt {5}}{\| \mathbf {x} \|} \mathbf {x} + \frac {1 0}{3} \mathbf {x}) - \frac {\sqrt {5}}{\| \mathbf {x} \|} \exp (- \sqrt {5} \| \mathbf {x} \|) (1 + \sqrt {5} \| \mathbf {x} \| + \frac {5}{3} \| \mathbf {x} \| ^ {2}) \mathbf {x} \\ = - \frac {5}{3} \exp (- \sqrt {5} \| \mathbf {x} \|) (1 + \sqrt {5} \| \mathbf {x} \|) \mathbf {x}. \\ \end{array}
$$
Since the Matérn kernel is isotropic, the  $\alpha_{i}$  and  $\beta_{i}$  as in Lemma 12 are the same across different coordinate  $i$ , and their values are
$$
\begin{array}{l} \alpha = \alpha_ {i} = \phi (2 h \mathbf {e} _ {i}) = \exp (- 2 \sqrt {5} h) (1 + 2 \sqrt {5} h + \frac {2 0}{3} h ^ {2}), \\ \beta = \beta_ {i} = \partial_ {i} \phi (- h \mathbf {e} _ {i}) = - \partial_ {i} \phi (h \mathbf {e} _ {i}) = \frac {5}{3} \exp (- \sqrt {5} h) (1 + \sqrt {5} h) h, \\ \end{array}
$$
In addition,  $-\partial_i^2\phi (\mathbf{0}) = \frac{5}{3}$ . By Lemma 12, we have
$$
\begin{array}{l} A _ {i i} \leq - \partial_ {i} ^ {2} \phi (\mathbf {0}) - \frac {2 \beta^ {2}}{(1 - \alpha) + \gamma} \\ = \frac {5}{3} - \frac {1 0 \exp (- 2 h) (1 + h) ^ {2} h ^ {2}}{3 (3 - \exp (- 2 h) (3 + 6 h + 4 h ^ {2})) + 9 \gamma}. \\ \end{array}
$$
Next, we approximate the exponential function  $\exp(-2h)$  by its Taylor polynomials. By Lemma 6, use the inequality  $\exp(-2h) \geq 1 - 2h$  for the numerator and the inequality  $\exp(-2h) \geq 1 - 2h + 2h^2 - \frac{4}{3} h^3$  for the denominator. Applying these two inequalities gives
$$
A _ {i i} \leq \frac {5}{3} - \frac {1 0 (1 - 2 h) (1 + h) ^ {2} h ^ {2}}{2 h ^ {2} (3 + 8 h ^ {3}) + 9 \gamma}.
$$
Let  $h = \gamma^{\frac{1}{4}}$  and thus  $\gamma = h^4$ . Then we have