SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {d} \rho (\theta) = \frac {1}{\sqrt {2 \pi}} \mathrm {e} ^ {- \frac {1}{2} \| \theta \| _ {2} ^ {2}} \mathrm {d} \theta
$$
and consider the tensor Hermite polynomials
$$
\Psi_ {\nu} (\theta) = \psi_ {\nu_ {1}} (\theta_ {1}) \dots \psi_ {\nu_ {d}} (\theta_ {d}), \qquad \nu = (\nu_ {i}) _ {i = 1} ^ {d} \in \mathbb {N} _ {0} ^ {d}, \theta = (\theta_ {i}) _ {i = 1} ^ {d} \in \mathbb {R} ^ {d}.
$$
Leveraging what we did in the one-dimensional case, we can then construct an orthonormal basis of the Sobolev space
$$
H _ {\rho} ^ {1} (\mathbb {R} ^ {d}) = \{f \in L _ {\rho} ^ {2} (\mathbb {R} ^ {d}): \partial_ {k} f \in L _ {\rho} ^ {2} (\mathbb {R} ^ {d}), k = 1, \ldots , d \}
$$
as  $\{\Phi_{\nu}\}_{\nu \in \mathbb{N}_0^d}$ , where
$$
\Phi_ {\nu} = \frac {\Psi_ {\nu}}{\sqrt {1 + \nu_ {1} + \ldots + \nu_ {d}}}, \quad \nu = (\nu_ {i}) _ {i = 1} ^ {d} \in \mathbb {N} _ {0} ^ {d}.
$$
We conclude by defining the approximation space  $\mathbb{F}$ . Fix an index  $p\in \mathbb{N}_0$  and let
$$
S = S _ {p} = \left\{\nu = \left(\nu_ {i}\right) _ {i = 1} ^ {d} \in \mathbb {N} _ {0} ^ {d}: \prod_ {i = 1} ^ {d} \left(\nu_ {i} + 1\right) \leq p + 1 \right\} \subset \mathbb {N} _ {0} ^ {d}. \tag {B.6}
$$
This is the so-called hyperbolic cross index set, which is particularly well suited for multivariate polynomial approximation [1, 37]. Using this, we define the linear approximation space
$$
\mathbb {F} = \operatorname {s p a n} \left\{\Phi_ {\nu}: \nu \in S \right\} \subset H _ {\rho} ^ {1} \left(\mathbb {R} ^ {d}\right).
$$
Notice that Assumption 2.5 holds for this choice of  $\mathbb{F}$ . Indeed,  $\mathbb{F}$  contains the constant function  $f(\theta) = 1, \forall \theta \in \mathbb{R}^d$ , which satisfies  $L(\theta)(f) = (1,0,\dots,0)^\top \neq 0$ .
B.3 Numerical solution of the regression problem
We now describe the numerical solution of the regression problem (2.2) in this case. Let  $\theta_1, \ldots, \theta_m \in \mathbb{R}^d$  be sample points and  $\nu: D \to \mathbb{R}$  be measurable and finite and positive almost everywhere. Then this problem takes the form
$$
\hat {f} \in \underset {f \in \mathbb {F}} {\operatorname {a r g m i n}} \frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\nu \left(\theta_ {i}\right)} \sum_ {k = 0} ^ {d} \left| y _ {i k} - \partial_ {k} f \left(\theta_ {i}\right) \right| ^ {2}, \tag {B.7}
$$
where  $y_{ik} = \partial_k f^* (\theta_i) + e_{ik}$  is the  $k$ th component of the  $i$ th measurement of  $f^{*}$ . For convenience, we assume the ordering  $S = \{\nu^1,\dots ,\nu^n\}$ . Let  $f\in \mathbb{F}$  be arbitrary and write  $f = \sum_{i = 1}^{n}c_{i}\Phi_{\nu^{i}}$  and  $c = (c_{i})_{i = 1}^{n}$ . Then
$$
\frac {1}{\sqrt {\nu (\theta_ {i}) m}} \partial_ {k} f (\theta_ {i}) = (A _ {k} c) _ {i},
$$
$$
A _ {k} = \left(\frac {1}{\sqrt {\nu (\theta_ {i}) m}} \partial_ {k} \Phi_ {\nu^ {j}} (\theta_ {i})\right) _ {i, j = 1} ^ {m, n}.
$$
Also, let
$$
b _ {k} = \left(\frac {1}{\sqrt {\nu (\theta_ {i}) m}} y _ {i k}\right) _ {i = 1} ^ {m}
$$
$$
A = \left[ \begin{array}{c} A _ {0} \\ \vdots \\ A _ {d} \end{array} \right], \quad b = \left[ \begin{array}{c} b _ {0} \\ \vdots \\ b _ {d} \end{array} \right].
$$
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\nu (\theta_ {i})} \sum_ {k = 0} ^ {d} | y _ {i k} - \partial_ {k} f (\theta_ {i}) | ^ {2} = \| A c - b \| _ {2} ^ {2}.
$$
Thus, the algebraic least-squares problem
$$
\hat {c} \in \operatorname * {a r g m i n} _ {c \in \mathbb {C} ^ {n}} \| A c - b \| _ {2} ^ {2}, \tag {B.8}
$$
is equivalent to (B.7), in the sense that every solution of this problem yields a solution  $\hat{f} = \sum_{i=1}^{n} \hat{c}_i \Phi_{\nu^i}$  of (B.7) and vice versa.
Notice (B.8) is an  $m(d + 1) \times n$  algebraic least-squares problem, which can be easily solved using standard linear algebra software once the matrix  $A$  has been computed. This is done using the relations (B.4) and (B.5).
B.4 Christoffel sampling
We now explain how to perform CS in this example. Directly sampling from the measure (2.7) is potentially difficult for this problem, since a simple characterization of the generalized Christoffel function  $K(\mathbb{F})(\theta)$  is lacking. However, we observe that
$$
\frac {1}{d + 1} \widetilde {K} (\mathbb {F}) (\theta) \leq K (\mathbb {F}) (\theta) \leq \widetilde {K} (\mathbb {F}) (\theta), \tag {B.9}
$$
$$
\widetilde {K} (\mathbb {F}) (\theta) = \sum_ {\nu \in S} \| L (\theta) \left(\Phi_ {\nu}\right) \| _ {\mathbb {Y}} ^ {2} = \sum_ {\nu \in S} \sum_ {k = 0} ^ {d} | \partial_ {k} \Phi_ {\nu^ {i}} (\theta) | ^ {2} = \sum_ {\nu \in S} \sum_ {k = 0} ^ {d} \frac {\left| \psi_ {\nu_ {k}} ^ {\prime} \left(\theta_ {k}\right) \right| ^ {2} \prod_ {\substack {l = 1 \\ l \neq k}} ^ {d} \left| \psi_ {\nu_ {l}} \left(\theta_ {l}\right) \right| ^ {2}}{1 + \| \nu \| _ {1}}. \tag{B.10}
$$
See Lemma E.1. Notice that
$$
\int_ {D} \widetilde {K} (\mathbb {F}) (\theta) \mathrm {d} \rho (\theta) = \sum_ {\nu \in S} \int_ {\mathbb {R} ^ {d}} \sum_ {k = 0} ^ {d} | \partial_ {k} \Phi_ {\nu} (\theta) | ^ {2} \mathrm {d} \rho (\theta) = \sum_ {\nu \in S} \| \Phi_ {\nu} \| _ {H _ {\rho} ^ {1} (\mathbb {R} ^ {d})} ^ {2} = n,
$$
since the functions  $\Phi_{\nu}$  are orthonormal. Therefore, we can use  $\widetilde{K} (\mathbb{F})(\theta)$  to construct a sampling measure as
$$
\mathrm {d} \tilde {\mu} (\theta) = \tilde {\nu} (\theta) \mathrm {d} \rho (\theta) = \frac {\widetilde {K} (\mathbb {F}) (\theta)}{n} \mathrm {d} \rho (\theta).
$$
Since  $\operatorname{ess}\sup_{\theta \sim \rho}\{K(\mathbb{F})(\theta) / \tilde{\nu} (\theta)\} \leq n$ , Corollary 4.3 implies that sampling from  $\tilde{\mu}$  yields optimal, log-linear sample complexity. As it is so closely related to the generalized Christoffel function (see (B.9)), we continue to refer to this as Christoffel sampling.
One can sample from  $\tilde{\mu}$  by observing that it is an additive mixture [34] of tensor-product probability measures, each of which can be sampled from efficiently [96]. However, we opt for a simpler approach based on [2, 92] involving finite grids and discrete measures. Let  $Z = \{z_{i}\}_{i = 1}^{N}\subset D$  grid of  $N\in \mathbb{N}$  points, with each point  $z_{i}$  drawn i.i.d. from the measure  $\rho$ . Then, we now replace the measure  $\rho$  by the finite measure
$$
\bar {\rho} = \frac {1}{N} \sum_ {i = 1} ^ {N} \delta_ {z _ {i}}. \tag {B.11}
$$
After doing this, the Christoffel function becomes a discrete function supported on the grid  $Z$ , which can be expressed as
$$
\widetilde {K} (\mathbb {F}) (z) = \sum_ {j = 1} ^ {n} \sum_ {k = 0} ^ {d} | \partial_ {k} \chi_ {j} (z) | ^ {2}, \quad z \in Z,
$$
where  $\{\chi_j\}_{j=1}^n$  is an orthonormal basis of  $\mathbb{F}$  with respect to the discrete  $H_{\bar{\rho}}^{1}(\mathbb{R})$ -inner product. Now define the matrix
$$
B = \left[ \begin{array}{c} B _ {0} \\ \vdots \\ B _ {d} \end{array} \right], \qquad B _ {k} = \left(\frac {1}{\sqrt {N}} \partial_ {k} \Phi_ {\nu^ {j}} (z _ {i})\right) _ {i, j = 1} ^ {N, n}.
$$
Consider the QR factorization  $B = QR$ , where  $Q = (q_{ij}) \in \mathbb{C}^{(d + 1)N \times n}$  has orthonormal columns and  $R \in \mathbb{C}^{n \times n}$  is upper triangular. Then the  $j$ th column of  $Q$  contains precisely the values  $\partial_k \chi_j(z_i)$ ,  $i = 1, \ldots, N$ ,  $k = 0, \ldots, d$ . Using this, we get
$$
\widetilde {K} (\mathbb {F}) (z _ {i}) = N \sum_ {j = 1} ^ {n} \sum_ {k = 0} ^ {d} | q _ {i + k N, j} | ^ {2}, \quad i = 1, \ldots , N.
$$
$$
\int \widetilde {K} (\mathbb {F}) (z) \mathrm {d} \bar {\rho} (z) = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \sum_ {k = 0} ^ {d} | q _ {i + k N, j} | ^ {2} = n,
$$
$$
\nu (z) = \frac {\widetilde {K} (\mathbb {F}) (z _ {i})}{n}.
$$
Hence, we can compute the function  $\widetilde{K} (\mathbb{F})$  over the grid  $Z$  and then use this to sample from the optimal measure  $\mu$ , which in this case is the discrete measure with  $\theta \sim \mu$  if
$$
\mathbb {P} (\theta = z _ {i}) = \frac {\widetilde {K} (\mathbb {F}) (z _ {i})}{n N}, \quad i = 1, \ldots , N.
$$
B.5 Additional experimental details
As described in Appendix B.4 we first draw a random grid from the measure  $\rho$ . In our experiments, we choose the number of grid points as  $N = 50,000$ . Note that this grid is drawn once, prior to any subsequent computations. In Fig. 1 we compute a test error as the relative Sobolev norm error over this grid. This is done using the following expression:
$$
E \left(f ^ {*}\right) = \frac {\left\| f ^ {*} - \hat {f} \right\| _ {H _ {\hat {\rho}} ^ {1} \left(\mathbb {R} ^ {d}\right)}}{\left\| f ^ {*} \right\| _ {H _ {\hat {\rho}} ^ {1} \left(\mathbb {R} ^ {d}\right)}} = \sqrt {\frac {\sum_ {k = 0} ^ {d} \frac {1}{N} \sum_ {i = 1} ^ {N} \left| \partial_ {k} \left(f ^ {*} - \hat {f}\right) \left(z _ {i}\right) \right| ^ {2}}{\sum_ {k = 0} ^ {d} \frac {1}{N} \sum_ {i = 1} ^ {N} \left| \partial_ {k} f ^ {*} \left(z _ {i}\right) \right| ^ {2}}}. \tag {B.12}
$$
Figure 4: CS for gradient-augmented polynomial regression. Plots of the average condition number  $\operatorname{cond}(A)$  versus  $m$  for MCS and CS for gradient-augmented polynomial regression for  $d = 2, 4, 8$  (left to right). In both cases, we use the scaling  $m = \lceil \max\{n, n \log(n)\} / (d + 1) \rceil$ .
The experiments shown in Fig. 1 proceed as follows. First, we generate a sequence of hyperbolic cross index sets (B.6) of orders  $1 \leq p_{1} < p_{2} < \dots < p_{L}$ . Second, for each  $l = 1, \ldots, L$ , we choose  $m = m_{l}$  according to the scaling
$$
m = \lceil \max  \{n, n \log (n) \} / (d + 1) \rceil , \tag {B.13}
$$
where  $n = n_{l} = |S_{p_{l}}|$ . Then, we draw  $m$  samples randomly using either MCS or CS (as described above), compute the approximation  $\hat{f}$  using (B.8) and then compute the error using (B.12). We then repeat this process for a total of  $T = 25$  trials.
The graphs in Fig. 1 and this appendix show the geometric mean (the main curve) and plus/minus one (geometric) standard deviation (the shaded region). The reason for using the geometric mean is because the errors are plotted in log-scale on the  $y$ -axis. See [1, Sec. A.1] for further rationale behind this choice of visualization.
These experiments were conducted on a 2012 MacBook Pro with a  $2.7\mathrm{GHz}$  Intel Core i7 CPU and 16 GB of DDR3-1600 RAM running MATLAB R2020b. Total runtime for these experiments was roughly three days.
B.6 Further experiments and discussion
Let  $0 \leq \alpha' \leq \beta' < \infty$  be the optimal constants in the empirical nondegeneracy condition (2.5). In this example, for a collection of sample points  $\theta_1, \ldots, \theta_m$ , these are given by
$$
\alpha^ {\prime} = \inf  \left\{\frac {1}{m} \sum_ {k = 0} ^ {d} \sum_ {i = 1} ^ {m} \frac {1}{\nu \left(\theta_ {i}\right)} \left| \partial_ {k} f \left(\theta_ {i}\right) \right| ^ {2}: f \in \mathbb {F}, \| f \| _ {\mathbb {X}} \leq 1 \right\}
$$
$$
\beta^ {\prime} = \sup  \left\{\frac {1}{m} \sum_ {k = 0} ^ {d} \sum_ {i = 1} ^ {m} \frac {1}{\nu (\theta_ {i})} | \partial_ {k} f (\theta_ {i}) | ^ {2}: f \in \mathbb {F}, \| f \| _ {\mathbb {X}} \leq 1 \right\}
$$
(here we recall that  $\mathbb{F} - \mathbb{F} = \mathbb{F}$ , since  $\mathbb{F}$  is a subspace). By expanding  $f\in \mathbb{F}$  in the orthonormal basis  $\{\Phi_{\nu}:\nu \in S\}$  we readily deduce that
$$
\alpha^ {\prime} = \sigma_ {\min } (A), \quad \beta^ {\prime} = \sigma_ {\max } (A).
$$
Thus, these constants can be computed numerically. Moreover, their ratio  $\beta' / \alpha' = \operatorname{cond}(A)$  is precisely the 2-norm condition number of  $A$ . In Fig. 4, we plot  $\operatorname{cond}(A)$  for both schemes using the same experimental setup as in Fig. 1. As predicted by our theory, CS leads to a bounded condition number for all  $m$ , that is at most roughly  $10^2$  in magnitude. However, MCS leads to an exploding condition number. This indicates that the scaling used Eq. (B.13) is not sufficient to ensure a good generalization bound, which is the reason for the poor performance of MCS in Fig. 1.
To examine this further, in Fig. 5 we perform the following experiment. For each value of  $n = n_l$ ,  $l = 1, \dots, L$ , we compute the minimum value of  $m = m_l$  such that  $\operatorname{cond}(A) \leq \operatorname{tol}$ , where  $\operatorname{tol} = 10$ . This is done as follows. We first set  $m = n / (d + 1)$  and compute  $\operatorname{cond}(A)$ . If  $\operatorname{cond}(A) > \operatorname{tol}$ , we increment  $m$  by one, draw new samples, re-compute  $\operatorname{cond}(A)$  and check whether  $\operatorname{cond}(A) \leq \operatorname{tol}$ . If not, we repeat this step. Finally, we repeat the whole process for a total of  $T = 25$  trials.
Figure 5: CS for gradient-augmented polynomial regression. Plots of minimum value of  $m$  that which yields a condition number  $\mathrm{cond}(A) \leq \mathrm{tol}$  for both MCS and CS, where  $\mathrm{tol} = 10$  and  $d = 2, 4, 8$  (left to right).
In Fig. 5, we show the results of this computation for MCS and CS. We plot both the arithmetic mean (the main curve) and plus/minus one standard deviation (the shaded region). As is evident, the CS scheme leads to a much less severe scaling. Further, as predicted by our main result, this scaling behaves as  $m \propto n \log(n)$ . On the other hand, MCS exhibits a severe scaling. When  $d = 2$ , for examples, one needs over 2000 sample points whenever  $n \geq 30$ . Notice that this exceedingly poor scaling for polynomial regression with MCS has been widely documented in the non-gradient augmented setting [57, 90, 114]. These experiments show gradient-augmented sampling also suffers from a similar effect.
As noted in Section 5, a limitation of this approach is that the benefit it delivers depends very much on the problem setting. For example, suppose we change the domain  $D$  to the bounded hypercube  $D = [-1,1]^d$  and consider polynomial regression using orthonormal polynomials with respect to the uniform probability measure on  $D$ . These polynomials can be constructed using tensor-products of the one-dimensional Legendre polynomials, much as was done in the previous case with the Hermite polynomials. In particular, analogous recurrence relations to (B.2) and (B.3) hold [28, Sec. 2.3], which allows for efficient construction of the least-squares matrix  $A$  once more. In Fig. 6 we display the approximation errors and condition numbers for MCS and CS for this problem. The comparison with Fig. 1 and Fig. 4 is striking. Especially in higher ( $d \geq 4$ ) dimensions MCS performs nearly as well as CS, both in terms of the approximation error and condition number. This behaviour is also reflected in scaling computations, which are shown in the third row of Fig. 6. In low dimensions, MCS suffers from a worse scaling than CS, but it is nowhere near as severe as in the case of unbounded domains. Moreover, as the dimension increases the scalings become much closer. Note that these observations are well known in the non-gradient augmented setting [57, 90].
B.7 Variation involving  $C > 1$
We conclude this appendix by briefly describing several variations on the gradient-augmented regression problem that take advantage of the multimodal  $(C > 1)$  formulation of the general sampling framework.
Partial gradient samples
In some of the aforementioned applications, one may only be able to afford fewer gradient samples than function evaluations [56, 103]. This can be handled as followed. Let  $C = 2$ ,  $m_{1}$  be the number of function only samples and  $m_{2}$  be the number of gradient-augmented samples. Then we define  $\mathbb{X} = H_{\rho}^{1}(\mathbb{R}^{d})$ ,  $D_{1} = D_{2} = \mathbb{R}^{d}$ ,  $\rho_{1} = \rho_{2} = \rho$ ,  $\mathbb{Y}_{1} = (\mathbb{C},\langle \cdot ,\cdot \rangle_{2})$ ,  $\mathbb{Y}_{2} = (\mathbb{C}^{d + 1},\langle \cdot ,\cdot \rangle_{2})$  and
$$
L _ {1} (\theta) (f ^ {*}) = c _ {0} f ^ {*} (\theta), \qquad L _ {2} (\theta) (f ^ {*}) = (c _ {k} \partial_ {k} f ^ {*} (\theta)) _ {k = 0} ^ {d},
$$
where  $c_0 = 1 / \sqrt{2}$  and  $c_k = 1$  otherwise. Then we have
$$
\sum_ {c = 1} ^ {C} \int_ {D _ {c}} \| L (\theta) (f ^ {*}) \| _ {\mathbb {Y} _ {c}} ^ {2} \mathrm {d} \rho_ {c} (\theta) = \int_ {D} | f ^ {*} (\theta) | ^ {2} \mathrm {d} \rho (\theta) + \sum_ {k = 0} ^ {d} \int_ {\mathbb {R} ^ {d}} | \partial_ {k} f ^ {*} (\theta) | ^ {2} \mathrm {d} \rho (\theta) = \| f ^ {*} \| _ {H _ {\rho} ^ {1} (\mathbb {R} ^ {d})} ^ {2}.
$$
Thus, empirical nondegeneracy holds (2.5) as before.
Figure 6: CS for gradient-augmented polynomial regression. Plots of the polynomial regression error (top row), condition number (middle row) and scalings (bottom row) for  $d = 1,2,4,8$  (left to right) when using the Legendre polynomials on  $D = [-1,1]^d$  instead of the Hermite polynomials on  $D = \mathbb{R}^d$ . These rows recreate Figs. 1, 4 and 5, respectively, for this setting.
Hierarchical sampling
A limitation of the standard CS is that it is not well suited to hierarchical approximation. Often in practice, rather than a single approximation, one may wish to compute a sequence of approximations  $\hat{f}_1, \hat{f}_2, \ldots$  it a nested sequence of subspaces  $\mathbb{F}_1 \subset \mathbb{F}_2 \subset \cdots$  using a nested sequence of samples  $\{\theta_i\}_{i=1}^{m_1} \subset \{\theta_i\}_{i=1}^{m_2} \subset \cdots$ . Standard CS is not well suited to this problem, since when the subspace changes from  $\mathbb{F}_k$  to  $\mathbb{F}_{k+1}$ , so does the generalized Christoffel function. Therefore the existing points are drawn from the wrong distribution for optimal sampling in the new subspace  $\mathbb{F}_{k+1}$ .
A means to overcome this problem was presented in [91] (see also [2]). It can be cast into our general framework using  $C = n = \dim(\mathbb{F})$  sampling operators. These works consider the unaugmented learning problem. However, it is readily extended to the gradient-augmented setting.
The basic idea is to define the  $i$ th sampling measure as
$$
\mathrm {d} \mu_ {i} (\theta) = \nu_ {i} (\theta) \mathrm {d} \rho (\theta), \quad \nu_ {i} (\theta) = \sum_ {k = 0} ^ {d} | \partial_ {k} \Phi_ {\nu^ {i}} (\theta) | ^ {2}, \quad i = 1, \dots , n. \tag {B.14}
$$
Notice that this is a probability measure, since the  $\Phi_{\nu^i}$  are orthonormal in  $H_{\rho}^{1}(\mathbb{R}^{d})$ , Then, assuming that  $m = kn$  for some  $k\in \mathbb{N}$ , we draw  $m_{i} = k$  samples i.i.d. according to  $\mu_{i}$  for each  $i = 1,\dots ,n$  giving  $m$  samples in total. We omit the full analysis of this case, but we remark in passing that this leads to near-optimal sample complexity. This is due, in essence, to the fact that
$$
\sum_ {i = 1} ^ {n} \nu_ {i} (\theta) = \widetilde {K} (\mathbb {F}) (\theta),
$$
where  $\widetilde{K} (\mathbb{F})$  is as in (B.10). The key point is that (B.14) is well suited to hierarchical approximation, since it draws  $k$  samples corresponding to each basis function  $\Phi_{\nu^i}$ . If the subspace  $\mathbb{F} = \operatorname {span}\{\Phi_{\nu^i}: i = 1,\ldots ,n\}$  is augmented to  $\mathbb{F}' = \operatorname {span}\{\Phi_{\nu^i}: i = 1,\dots ,n'\}$ , where  $n^{\prime}\geq n$ , then we can recycle the existing samples, and draw additional samples for the new basis functions  $\Phi_{\nu^i}, i = n + 1,\dots ,n'$ . See [2, 91] for further details.
C Fourier imaging with generative models
In this appendix, we describe the example considered in Section 3.2. We consider a discrete imaging scenario where the object to recover is a  $d$ -dimensional image of size  $n \times \dots \times n$ . In this paper, we consider 3D imaging, i.e.,  $d = 3$ .
Let  $N = n^d$  and  $f^{*}\in \mathbb{C}^{N}$  be the complex-valued vectorized image. Let  $F\in \mathbb{C}^{N\times N}$  be the matrix of the  $d$ -dimensional Discrete Fourier Transform (DFT) of length  $n$  with normalization  $F^{*}F = I$ . In (subsampled) Fourier imaging, one selects  $m\leq N$  frequencies (typically,  $m\ll N$ ), which correspond to  $m$  rows of  $F$ . Let  $\Omega \subseteq \{1,\dots ,N\}$ ,  $|\Omega | = m$ , be the set of frequencies sampled. Then the noisy measurements of the unknown image  $f^{*}$  take the form
$$
P _ {\Omega} F f ^ {*} + e,
$$
where  $e \in \mathbb{C}^m$  is noise and  $P_{\Omega} \in \mathbb{C}^{m \times N}$  is the row selector matrix, i.e.,  $P_{\Omega}F$  selects the rows of  $F$  corresponding to the indices in  $\Omega$ .
C.1 Formulation in terms of the general sampling framework
Depending on the imaging modality, it may be possible to select individual frequencies according to some sampling strategy. However, in applications such as MRI, this may not be possible. Typically, the scanner is only allowed to sample along piecewise smooth curves in frequency space [80, 89]. Fortunately, as we now describe, both models can be cast in our general sampling framework.
Let  $\mathbb{X} = \mathbb{X}_0 = \mathbb{C}^N$  with the Euclidean inner product and norm and  $C = 1$ . Let  $\Delta_1, \ldots, \Delta_M \subseteq \{1, \ldots, N\}$  be a partition of  $\{1, \ldots, N\}$ , i.e.,  $\cup_{i=1}^{M} \Delta_i = \{1, \ldots, N\}$ . For convenience, we assume the  $\Delta_i$  have the same cardinality, i.e.,  $|\Delta_i| = p$ ,  $\forall i$ . Then we let  $D = \{1, \ldots, M\}$ ,  $\rho$  be the uniform probability measure on  $D$  and  $\mathbb{Y} = \mathbb{C}^p$  with the Euclidean inner product. Next, we define the sampling operator
$$
L: D \to \mathcal {B} (\mathbb {C} ^ {N}, \mathbb {C} ^ {p}),
$$
$$
L (i) (x) = \sqrt {M} \left(\left(F x\right) _ {j}\right) _ {j \in \Delta_ {i}} \in \mathbb {C} ^ {p}, \quad \forall x \in \mathbb {C} ^ {N}. \tag {C.1}
$$
In other words,  $L(i)$  is the linear operator that outputs the Fourier transform of an image  $x$  at the frequencies in the  $i$ th partition element  $\Delta_i$ . Observe that Assumption 2.4 holds for this example with  $\alpha = \beta = 1$ . Indeed, for any  $x \in \mathbb{C}^N$ , we have
$$
\int_ {D} \| L (\theta) (x) \| ^ {2} \mathrm {d} \rho (\theta) = \frac {1}{M} \sum_ {i = 1} ^ {M} \| L (i) (x) \| _ {\mathbb {Y}} ^ {2} = \sum_ {i = 1} ^ {M} \sum_ {j \in \Delta_ {i}} | (F x) _ {j} | ^ {2} = \| F x \| _ {2} ^ {2} = \| x \| _ {2} ^ {2}. \tag {C.2}
$$
With this in hand, our goal now is to design a suitable sampling measure  $\mu$  on  $D = \{1,\dots ,M\}$ , so that each random draw from  $\mu$  corresponds to a sample of frequencies from the corresponding partition element. Let us first describe two examples.
Example C.1 (Sampling individual frequencies) Suppose the modality allows for sampling one frequency at a time. Here, we let  $M = N$  and define  $\Delta_i = \{i\}$ ,  $i = 1, \dots, N$ .
We consider Example C.1 later in Fig. 8. Fig. 2 in the main paper is, however, based on the following example.
Example C.2 (Sampling horizontal lines in  $k$ -space) Phase encoding is a common sampling strategy in MRI. Here one fully samples in the first  $k$ -space direction and subsamples in the others. Thus, the scanner traverses a piecewise smooth trajectory in  $k$ -space. Mathematically, this can be formulated as follows. Let  $d = 3$  for concreteness (other values of  $d$  can be easily handled),  $N = n^3$  and  $M = n^2$ . 3D frequencies are naturally enumerated over the set of multi-indices  $\{1, \ldots, n\}^3$ . Now consider a partition  $\Delta_{i_1 i_2}^\prime$ ,  $i_1, i_2 = 1, \ldots, n$ , of this set defined by
$$
\Delta_ {i _ {1} i _ {2}} ^ {\prime} = \{(j, i _ {1}, i _ {2}): j \in \{1, \dots , n \} \}.
$$
Thus,  $\Delta_{i_1i_2}'$  consists of  $k$ -space values along a horizontal line with  $y-z$  coordinate  $(i_1,i_2)$ . We now let  $\Delta_{i_1i_2} \subset \{1,\dots,N\}$  be the image of  $\Delta_{i_1i_2}'$  with respect to whatever ordering is used to convert triples  $(i,j,k) \in \{1,\dots,n\}^3$  to single indices in  $\{1,\dots,N\}$  (in our experiments, we use lexicographical ordering). Then the sets  $\Delta_{i_1i_2}$  define a partition of  $\{1,\dots,N\}$ , as required.