SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now note that the least-squares problem in Eq. (13) has at least one optimal solution  $\tilde{g}$  which is in the eigenspace of the operator  $P P^{*}$ . More specifically, there exists a vector  $\mathbf{z} \in \mathbb{R}^s$  such that  $\tilde{g} = \mathbf{P} \cdot \mathbf{z}$  is an optimal solution for Eq. (13). Therefore, we can focus on finding this optimal solution by solving the following least-squares problem
$$
\boldsymbol {z} \in \arg \min  _ {\boldsymbol {x} \in \mathbb {R} ^ {s}} \| \boldsymbol {P} ^ {*} \boldsymbol {P} \boldsymbol {x} - \boldsymbol {f} \| _ {2} ^ {2},
$$
and then letting  $\tilde{g} = P\cdot z$ . This  $\tilde{g}$  is guaranteed to be an optimal solution for Eq. (13), thus it satisfies Eq. (14). We solve the above least-squares problem using the kernel trick. In fact we show that  $P^{*}P$  is equal to the kernel matrix  $K$  computed in line 4 of Algorithm 1. To see why, note that for any
$i,j\in [s]$  we have,
$$
\begin{array}{l} [ \boldsymbol {P} ^ {*} \boldsymbol {P} ] _ {i, j} = \left\langle \sum_ {\ell = 0} ^ {q} \frac {\alpha_ {\ell , d}}{\sqrt {s} \cdot | \mathbb {S} ^ {d - 1} |} \cdot P _ {d} ^ {\ell} (\langle w _ {i}, \cdot \rangle), \sum_ {\ell = 0} ^ {q} \frac {\alpha_ {\ell , d}}{\sqrt {s} \cdot | \mathbb {S} ^ {d - 1} |} \cdot P _ {d} ^ {\ell} (\langle w _ {j}, \cdot \rangle) \right\rangle_ {\mathbb {S} ^ {d - 1}} \\ = \sum_ {\ell = 0} ^ {q} \sum_ {\ell^ {\prime} = 0} ^ {q} \frac {\alpha_ {\ell , d} \alpha_ {\ell^ {\prime} , d}}{s \cdot | \mathbb {S} ^ {d - 1} | ^ {2}} \cdot \left\langle P _ {d} ^ {\ell} (\langle w _ {i}, \cdot \rangle), P _ {d} ^ {\ell^ {\prime}} (\langle w _ {j}, \cdot \rangle) \right\rangle_ {\mathbb {S} ^ {d - 1}} \\ = \sum_ {\ell = 0} ^ {q} \sum_ {\ell^ {\prime} = 0} ^ {q} \frac {\alpha_ {\ell , d} \alpha_ {\ell^ {\prime} , d}}{s \cdot | \mathbb {S} ^ {d - 1} |} \cdot \underset {v \sim \mathcal {U} (\mathbb {S} ^ {d - 1})} {\mathbb {E}} \left[ P _ {d} ^ {\ell} \left(\langle w _ {i}, v \rangle\right) \cdot P _ {d} ^ {\ell^ {\prime}} \left(\langle w _ {j}, v \rangle\right) \right] \\ = \sum_ {\ell = 0} ^ {q} \frac {\alpha_ {\ell , d}}{s \cdot | \mathbb {S} ^ {d - 1} |} \cdot P _ {d} ^ {\ell} \left(\langle w _ {i}, w _ {j} \rangle\right) = \boldsymbol {K} _ {i, j}, \\ \end{array}
$$
where the fourth line above follows from Lemma 1. Therefore, we are interested in the optimal solution of the following least-squares problem
$$
\boldsymbol {z} \in \arg \min  _ {\boldsymbol {x} \in \mathbb {R} ^ {s}} \| \boldsymbol {K} \boldsymbol {x} - \boldsymbol {f} \| _ {2} ^ {2}.
$$
The least-squares solution to the above problem is  $\pmb{z} = \pmb{K}^{\dagger}\pmb{f}$  which is exactly what is computed in line 6 of the algorithm. Now note that, the function  $\tilde{g} = \pmb{P} \cdot \pmb{z}$  satisfies Eq. (14). Because  $\tilde{g} = \pmb{P} \cdot \pmb{z} \in \mathcal{H}^{(q)}(\mathbb{S}^{d - 1})$  and because  $\mathcal{K}_d^{(q)}$  is an orthonormal projection operator into  $\mathcal{H}^{(q)}(\mathbb{S}^{d - 1})$ , we have  $\mathcal{K}_d^{(q)} \cdot \tilde{g} = \tilde{g} = \pmb{P} \cdot \pmb{z}$ . This together with Eq. (14) imply that,
$$
\left\| \boldsymbol {P} \cdot \boldsymbol {z} - f \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2} \leq (1 + \varepsilon) \cdot \min  _ {g \in L ^ {2} (\mathbb {S} ^ {d - 1})} \left\| \mathcal {K} _ {d} ^ {(q)} g - f \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2}.
$$
Now if we invoke Claim 2 with  $C = 1 + \varepsilon$  on the above inequality we find that,
$$
\left\| \boldsymbol {P} \cdot \boldsymbol {z} - f ^ {(q)} \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2} \leq \varepsilon \cdot \left\| f ^ {(q)} - f \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2}.
$$
Finally, one can easily see that the function  $y \in \mathcal{H}^{(q)}(\mathbb{S}^{d - 1})$  that Algorithm 1 outputs in line 7 is exactly equal to  $y = P \cdot z$ . This completes the accuracy bound of the theorem.
Runtime and Sample Complexity. these bounds follow from observing that:
-  $s \cdot d$  time is needed to generate  $w_{1}, w_{2}, \ldots, w_{s}$  in line 3 of the algorithm. To do this, we first generate random Gaussian points in  $\mathbb{R}^{d}$  and then project then onto  $\mathbb{S}^{d - 1}$  by normalizing them.
-  $s^2 \cdot d$  operations are needed to form the kernel matrix  $K$  in line 4 of the algorithm.
-  $s$  queries to function  $f$  are needed to form the samples vector  $\pmb{f}$  in line 5 of the algorithm.
-  $s^\omega$  time is needed to compute the least-squares solution  $z = K^\dagger f$  in line 6 of the algorithm.
-  $s \cdot d$  operations are needed to evaluate the output function  $y(\sigma)$  in line 7 of the algorithm.
This completes the proof of Theorem 5.
E Lower Bound: Claims and Lemmas
In this section we prove the Claims and Lemmas used in our lower bound analysis for proving Theorem 6.
Claim 4. Given the random input  $f = \sum_{\ell=0}^{q} \mathbf{Y}_{\ell} \cdot v^{(\ell)}$  generated as described in Section 4, to solve Problem 2, an algorithm must return a function  $\tilde{f}^{(q)} \in \mathcal{H}^{(q)}(\mathbb{S}^{d-1})$  such that  $\| \tilde{f}^{(q)} - f \|_{\mathbb{S}^{d-1}}^2 = 0$ .
Proof. Note that Problem 2 requires recovering a function  $\tilde{f}^{(q)}\in \mathcal{H}^{(q)}\left(\mathbb{S}^{d - 1}\right)$  such that:
$$
\left\| \tilde {f} ^ {(q)} - f ^ {(q)} \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2} \leq \varepsilon \cdot \left\| f ^ {(q)} - f \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2}, \tag {15}
$$
where  $f^{(q)} = \mathcal{K}_d^{(q)}f$ . Using the definition of the input function  $f = \sum_{\ell = 0}^{q}\mathbf{Y}_{\ell}\cdot v^{(\ell)}$ , we can write,
$$
\begin{array}{l} f ^ {(q)} = \mathcal {K} _ {d} ^ {(q)} f = \sum_ {\ell = 0} ^ {q} \mathcal {K} _ {d} ^ {(q)} \cdot \mathbf {Y} _ {\ell} \cdot v ^ {(\ell)} \\ = \sum_ {\ell = 0} ^ {q} \left(\sum_ {\ell^ {\prime} = 0} ^ {q} \boldsymbol {Y} _ {\ell^ {\prime}} \boldsymbol {Y} _ {\ell^ {\prime}} ^ {*}\right) \cdot \boldsymbol {Y} _ {\ell} \cdot v ^ {(\ell)} \\ = \sum_ {\ell = 0} ^ {q} \boldsymbol {Y} _ {\ell} \cdot v ^ {(\ell)} = f, \\ \end{array}
$$
where the equality in the second line above follows from Eq. (10) and the addition theorem in Theorem 3, and the third line follows because the operator  $\mathbf{Y}_{\ell}$  has orthonormal columns and thus  $\mathbf{Y}_{\ell'}^*\mathbf{Y}_{\ell} = I_{\alpha_{\ell,d}} \cdot \mathbb{1}_{\{\ell=\ell'\}}$ . Therefore, plugging this into Eq. (15) gives,
$$
\left\| \tilde {f} ^ {(q)} - f \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2} = \left\| \tilde {f} ^ {(q)} - f ^ {(q)} \right\| _ {\mathbb {S} ^ {d - 1}} ^ {2} \leq \varepsilon \cdot \| f ^ {(q)} - f \| _ {\mathbb {S} ^ {d - 1}} ^ {2} = \varepsilon \cdot \| f - f \| _ {\mathbb {S} ^ {d - 1}} ^ {2} = 0.
$$
Proof. By Claim 4, the output of the algorithm that solves Problem 2, satisfies  $\left\| \tilde{f}^{(q)} - f\right\|_{\mathbb{S}^{d - 1}}^2 = 0$ . Therefore, by orthonormality of the columns of the operator  $Y_{\ell}$ , we can write,
$$
\boldsymbol {Y} _ {\ell} ^ {*} \tilde {f} ^ {(q)} = \boldsymbol {Y} _ {\ell} ^ {*} f + \boldsymbol {Y} _ {\ell} ^ {*} (\tilde {f} ^ {(q)} - f) = \sum_ {\ell^ {\prime} = 0} ^ {q} \boldsymbol {Y} _ {\ell} ^ {*} \boldsymbol {Y} _ {\ell^ {\prime}} \cdot v ^ {(\ell^ {\prime})} = v ^ {(\ell)}.
$$