SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Many machine learning applications involve regressing a smooth, multivariate function using a model class consisting of algebraic polynomials. Often, the training data consists of function values. Yet, as noted in Section 1.1, there are many settings where one can acquire both function values and values of its gradient. In this case, the training data takes the form  $\{(\theta_i,f(\theta_i),\nabla f(\theta_i))\}_{i = 1}^m\subset \mathbb{R}^d\times \mathbb{R}^{d + 1}$ . As we explain in Appendix B.1, this problem fits into our framework with  $C = 1$ , where  $\rho$  is the standard Gaussian measure on  $\mathbb{R}^d$ ,  $\mathbb{X} = H_{\rho}^{1}(\mathbb{R}^{d})$  is the Sobolev space of weighted square-integrable functions with weighted square-integrable first-order derivatives and  $\mathbb{Y} = \mathbb{R}^{d + 1}$  with the Euclidean inner product. It therefore provides a first justification for allowing non-scalar valued sampling operators. Note that several important variations on this problem also fit into our general framework with  $C > 1$ . See Appendix B.7.
We consider learning a function from such training data in a sequence of nested polynomial spaces  $\mathbb{F}_1\subset \mathbb{F}_2\subset \dots$  . These spaces are based on hyperbolic cross index sets, which are particularly useful in multivariate polynomial regression tasks [1, 37]. See Appendix B.2 for the formal definition.
In Fig. 1 we compare gradient-augmented polynomial regression with CS versus MCS from the Gaussian measure  $\rho$ . See Appendices B.3-B.5 for details on how we implement CS in this case. CS gives a dramatic improvement over MCS. CS is theoretically near-optimal in the sense of Theorem 2.7 - i.e., it provably yields a log-linear sample complexity - since the approximation spaces are linear subspaces in this example. On the other hand, MCS fails, with the error either not decreasing or diverging. The reason is the log-linear scaling, which is not sufficient to ensure a generalization error bound of the form (2.9) for MCS. As we demonstrate in Appendix B.6, MCS requires a much more severe scaling of  $m$  with  $n$  to ensure a generalization bound of this type.
3.2 MRI reconstruction using generative models
Reconstructing an image from measurements is a fundamental task in science, engineering and industry. In many applications – in particular, medical imaging modalities such as MRI – one wishes to reduce the number of measurements while still retaining image quality. As noted, techniques based on DL have recently led to significant breakthroughs in image recovery tasks. One promising approach involves using generative models [16, 20, 67]. First, a generative model is trained on a database of relevant images, e.g., brain images in the case of MRI. Then the image recovery problem is formulated as a regression problem, such as (2.2), where  $\mathbb{F}$  is the range of the generative model. Note that in this problem, the training data consists of a finite set of frequencies and the corresponding values of the Fourier transform of the unknown image.
As we explain in Appendices C.1-C.3, this problem fits into our general framework. In Fig. 2 we demonstrate the efficacy of CS for Fourier imaging with generative models. In this example, the
Figure 2: CS for MRI reconstruction using generative models. Plots of (left) the average PSNR vs. image number of the 3-dimensional brain MR image for both CS and MCS methods at  $0.125\%$ ,  $0.375\%$ , and  $0.625\%$  sampling percentages, (middle) the average PSNR computed over the frames 30 to 90 of the image vs. sampling percentage for both CS and MCS methods, and (right) the empirical function  $\tilde{K}$  used for the CS procedure.
generative model was trained on a database of 3D MRI brain images (see Appendix C.5). This experiment simulates a 3D image reconstruction problem in MRI, where the measurements are samples of the Fourier transform of the unknown image taken along horizontal lines in  $k$ -space (a sampling strategy commonly known as phase encoding). The active learning problem involves judiciously choosing  $m$  horizontal lines in  $k$ -space to enhance the generalization performance of the learning procedure. In Fig. 2, we compare the average Peak Signal-to-Noise Ratio (PSNR) versus frame (i.e., 2D image slice) number for CS versus MCS (i.e., uniform random sampling) for reconstructing an unknown image. We observe a significant improvement, especially in the challenging regime where the sampling percentage (the ratio of the number of measurements to the image size) is low.
This example lies close to our main theorem, but is not fully covered by it. See Appendix C.3. The space  $\mathbb{F}$  (the range of a generative model) is not a finite union of subspaces. However, it is known that certain generative models (namely, those based on ReLU activation functions) are subsets of unions of subspaces of controllable size [16]. Further, we do not sample exactly from (2.7) in this case, but rather an empirical approximation to it (see Appendix C.4). Nevertheless, our experiments show a significant performance gain from CS in this case, despite it lying strictly outside of our theory.
This application justifies the presence of arbitrary (i.e., non-pointwise) sampling operators in our framework. It is another instance of non-scalar valued sampling operators, since each measurement is a vector of frequencies values along a horizontal line in  $k$ -space. Fig. 2 considers  $C = 1$  sampling operators. However, as we explain Appendix C.7, the important extension of this setup to parallel MRI (which is standard in clinical practice) can be formulated in our framework with  $C > 1$ .
3.3 Adaptive sampling for solving PDEs with PINNs
In our final example, we apply this framework to solving PDEs via Physics-Informed Neural Networks (PINNs). PINNs have recently shown great potential and garnered great interest for approximating solutions of PDEs [42, 105, 112]. It is typical in PINNs to generate samples via Monte Carlo Sampling (MCS). Yet, this suffers from a number of limitations, including low accuracy.
We use this general framework combined with the adaptive basis viewpoint [35] to devise a new adaptive sampling procedure for PINNs. Here, a Deep Neural Network (DNN) with  $n$  nodes in its penultimate layer is viewed as an element of the linear subspace spanned the  $n$  functions defined by this layer's nodes. Our method then proceeds as follows. First, we use an initial set of  $m = m_{1}$  samples and train the corresponding PINN  $\Psi = \Psi_{1}$ . Then, we use the adaptive basis viewpoint to construct a subspace  $\mathbb{F}_1$  with  $\Psi_{1} \in \mathbb{F}_{1}$ . Next, we draw  $m_{2} - m_{1}$  samples using CS for the subspace  $\mathbb{F}_1$ , and use the set of  $m = m_{2}$  samples to train a new PINN  $\Psi = \Psi_{2}$ , using the weights and biases of  $\Psi_{1}$  as the initialization. We then repeat this process, alternating between generating new samples via CS and retraining the network, to obtain a sequence of PINNs  $\Psi_{1}, \Psi_{2}, \ldots$  approximating the solution of the PDE. We term this procedure Christoffel Adaptive Sampling (CAS). See Appendices D.1-D.4 for further information.
Figure 3: CAS for solving PDEs with PINNs. Plots of (left) the average test error  $E(u^{*})$  versus the number of samples used for training tanh and sigmoid  $5 \times 50$  DNNs using CAS (solid line) and MCS (dashed line), (middle) the samples generated by the CAS for the tanh  $5 \times 50$  DNN and (right) the samples generated by CAS for the sigmoid  $5 \times 50$  DNN. The colour indicates the density of the points.
In Fig. 3 we show that this procedure gives a significant benefit over MCS in terms of the number of samples needed to reach a given accuracy. See Appendix D.5 for further details on the experimental setup. Note that both approaches use the same DNN architecture and are trained in exactly the same way (optimizer, learning rate schedule, number of epochs). Thus, the benefit is fully derived from the sampling strategy. The PDE considered (Burger's equations) exhibits shock formation as time increases. As can be seen in Fig. 3, CAS adapts to the unknown PDE solution by clustering samples near this shock, to better recover the solution than MCS.
As we explain in Appendix D.2, this example also justifies  $C > 1$  in our general framework, as we have three sampling operators related to the PDE and its initial and boundary conditions. We note, however, that this example falls outside our theoretical analysis, since the sampling operator stemming from the PDE is nonlinear and the method is implemented in an adaptive fashion. In spite of this, however, we still see a nontrivial performance boost from the CS-based scheme.
In this section, we present our theoretical analysis. Proofs can be found in Appendix E.
4.1 Sample complexity
We first establish a sufficient condition for empirical nondegeneracy (2.5) to hold in terms of the numbers of samples  $m_{c}$  and the generalized Christoffel functions of suitable spaces.
Definition 4.1 (Subspace covering number). Let  $(\mathbb{X},\| \cdot \|_{\mathbb{X}})$  be a quasisemi-normed vector space,  $\mathbb{F}$  be a subset of  $\mathbb{X}$ ,  $n\in \mathbb{N}_0$  and  $t\geq 0$ . A collection of subspaces  $\mathbb{F}_1,\ldots ,\mathbb{F}_d$  of  $\mathbb{X}$  of dimension at most  $n$  is a  $(\| \cdot \|_{\mathbb{X}},n,t)$ -subspace covering of  $\mathbb{F}$  if for every  $f\in \mathbb{F}$  there exists a  $j\in \{1,\dots ,d\}$  and a  $f_j\in \mathbb{F}_j\cap \mathbb{F}$  such that  $\| f - f_j\|_{\mathbb{X}}\leq t$ . The subspace covering number  $\mathcal{N}(\mathbb{F},\| \cdot \|_{\mathbb{X}},n,t)$  of  $\mathbb{F}$  is the smallest  $d$  such that there exists a  $(\| \cdot \|_{\mathbb{X}},n,t)$ -subspace covering of  $\mathbb{F}$  consisting of  $d$  subspaces.
In this definition, we consider a zero-dimensional subspace as a singleton  $\{x\} \subseteq \mathbb{X}$ . Therefore, the subspace covering number  $\mathcal{N}(\mathbb{F},\| \cdot \|_{\mathbb{X}},0,t)$  is precisely the classical covering number  $\mathcal{N}(\mathbb{F},\| \cdot \|_{\mathbb{X}},t)$ . We also remark that if  $\mathbb{F} = \cup_{j = 1}^{d}\mathbb{F}_{j}$  is itself a union of  $d$  subspaces of dimension at most  $n$ , then  $\mathcal{N}(\mathbb{F},\| \cdot \|_{\mathbb{X}},n^{\prime},t) = d$  for any  $t\geq 0$  and  $n^\prime \geq n$ .
We now also require the following notation. If  $\mathbb{F} \subseteq \mathbb{X}$ , we set  $\mathcal{B}(\mathbb{F}) = \{f / \| f\|_{\mathbb{X}} : f \in \mathbb{F} \backslash \{0\}\}$ .
Theorem 4.2 (Sample complexity for empirical nondegeneracy). Consider the setup of Section 2.1. Let  $\mathbb{F} \subseteq \mathbb{X}_0$ ,  $n \in \mathbb{N}_0$ ,  $0 < \epsilon, \delta < 1$  and  $\mathbb{F}_1, \ldots, \mathbb{F}_d$  be a  $(\|\cdot\|, n, t)$ -subspace covering of  $\mathcal{B}(\mathbb{F} - \mathbb{F})$ , where  $t = \sqrt{\alpha\epsilon/2}$  and  $\|\cdot\|^2 = \sum_{c=1}^{C} \operatorname{ess~sup}_{\theta_c \sim \rho_c} \left\{\|L_c(\theta_c)(\cdot)\|_{\mathbb{Y}_c}^2 / \nu(\theta_c)\right\}$ . Suppose that
$$
m _ {c} \geq c _ {\epsilon / 2} \cdot \alpha^ {- 1} \cdot \operatorname {e s s} \sup  _ {\theta \sim \rho_ {c}} \left\{K _ {c} (\widetilde {\mathbb {F}}) (\theta) / \nu_ {c} (\theta) \right\} \cdot \log (2 d \max  \{n, 1 \} / \delta), \quad \forall c = 1, \dots , C, \tag {4.1}
$$
where  $c_{\epsilon /2}$  is as in (E.6) and  $\widetilde{\mathbb{F}} = \cup_{i = 1}^{d}\mathbb{F}_{i}$ . Then (2.5) holds with probability at least  $1 - \delta$ .
This theorem gives the desired condition (4.1) for empirical nondegeneracy (2.5). It is interesting to understand when this condition can be replaced by one involving the function  $K_{c}$  evaluated over  $\mathbb{F} - \mathbb{F}$  rather than its cover  $\widetilde{\mathbb{F}}$ . We now examine two important cases where this is possible.
Corollary 4.3 (Sample complexity for unions of subspaces). Suppose that  $\mathbb{F} - \mathbb{F} = \cup_{j=1}^{d} \mathbb{F}_j$  is a union of  $d$  subspaces of dimension at most  $n \in \mathbb{N}$ . Then (4.1) is equivalent to
$$
m _ {c} \geq c _ {\epsilon / 2} \cdot \alpha^ {- 1} \cdot \operatorname * {e s s} _ {\theta \sim \rho_ {c}} \left\{K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) / \nu_ {c} (\theta) \right\} \cdot \log (2 n d / \delta), \quad c = 1, \ldots , C.
$$
Corollary 4.4 (Sample complexity for classical coverings). Consider Theorem 4.2 with  $n = 0$ . Then (4.1) is implied by the condition
$$
m _ {c} \geq c _ {\epsilon / 2} \cdot \alpha^ {- 1} \cdot \operatorname * {e s s} _ {\theta \sim \rho_ {c}} \left\{K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) / \nu_ {c} (\theta) \right\} \cdot \log (2 d / \delta), \quad c = 1, \ldots , C.
$$
Remark 4.5 Theorem 4.2 is formulated generally in terms of subspace coverings. This is done so that the scenarios covered in Corollaries 4.3 and 4.4 are both straightforward consequences. Our main examples in Section 3 are based on the (unions of) subspaces case. While this assumption is relevant for these and many other examples, there are some key problems that do not satisfy it. For example, in low-rank matrix (or tensor) recovery [27, 38, 108, 117], the space  $\mathbb{F}$  of rank- $r$  matrices (or tensors) is not a finite union of finite-dimensional subspaces. But tight bounds for its covering number are known [26, 106], meaning it fits within the setup of Corollary 4.4.
4.2 Christoffel sampling
Theorem 4.2 reduces the question of identifying optimal sampling measures to that of finding weight functions  $\nu_{c}$  that minimize the corresponding essential supremum in (4.1). The following lemma shows that this is minimized by choosing  $\nu_{c}$  proportional to the generalized Christoffel function.
Lemma 4.6 (Optimal sampling measure). Let  $L$ ,  $\mathbb{F}$ ,  $K(\mathbb{F})$  and  $\kappa(\mathbb{F}) < \infty$  be as in Definition 2.6. Suppose that for almost every  $\theta \in D$  there exists a  $f \in \mathbb{F}$  such that  $L(\theta)(f) \neq 0$ . Then
$$
\operatorname *{ess  sup}_{\theta \sim \rho}\{K(\mathbb{F})(\theta) / \nu (\theta)\} \geq \kappa (\mathbb{F})
$$
for any measurable  $\nu : D \to \mathbb{R}$  that is positive almost anywhere and satisfies  $\int_{D} \nu(\theta) \, \mathrm{d}\rho(\theta) = 1$ . Moreover, this optimal value is attained by the function  $\nu^{\star}(\theta) = K(\mathbb{F})(\theta) / \kappa(\mathbb{F})$ ,  $\theta \in D$ .
In view of this lemma, to optimize the sample complexity bound (4.1) we choose sampling measures
$$
\mathrm {d} \mu_ {c} ^ {\star} (\theta) = K _ {c} (\widetilde {\mathbb {F}}) (\theta) \mathrm {d} \rho_ {c} (\theta) / \kappa_ {c} (\mathbb {F}), \quad c = 1, \dots , C. \tag {4.2}
$$
As noted, we term this Christoffel Sampling (CS). This leads to a sample complexity bound
$$
m _ {c} \geq c _ {\epsilon / 2} \cdot \alpha^ {- 1} \cdot \kappa_ {c} (\widetilde {\mathbb {F}}) \cdot \log (2 d \max  \{n, 1 \} / \delta), \quad c = 1, \dots , C. \tag {4.3}
$$
In the case of subspaces (or union thereof with small  $d$ ), this approach leads to a near-optimal sample complexity bound for the total number of measurements  $m \coloneqq m_1 + \dots + m_C$ .
Corollary 4.7 (Near-optimal sampling for unions of subspaces). Suppose that  $\mathbb{F} - \mathbb{F} = \cup_{j=1}^{d} \mathbb{F}_{j}$  is a union of  $d$  subspaces of dimension at most  $n \in \mathbb{N}$ . Then  $\sum_{c=1}^{C} \kappa_{c}(\mathbb{F} - \mathbb{F}) \leq \beta dn$ . Therefore, choosing the sampling measures as in (4.2) with  $\widetilde{\mathbb{F}} = \mathbb{F} - \mathbb{F}$  and the number of samples  $m_{c} \leq c_{\epsilon/2} \cdot \alpha^{-1} \cdot \kappa_{c}(\mathbb{F} - \mathbb{F}) \cdot \log(2nd/\delta)$  leads to the overall sample complexity bound
$$
m \leq c _ {\epsilon / 2} \cdot (\beta / \alpha) \cdot n \cdot d \cdot \log (2 n d / \delta).
$$
We note in passing that  $\sum_{c=1}^{C} \kappa_c(\mathbb{F} - \mathbb{F}) \geq \alpha n / p$  whenever  $\dim(\mathbb{Y}_c) \leq p \in \mathbb{N}, \forall c$ . See Theorem E.4. Therefore, the sample complexity bound is at least  $\mathcal{O}_{\epsilon, \alpha, p}(n \log (nd / \delta))$ .
It is worth comparing these bounds with MCS. Suppose that the  $\rho_c$ 's are probability measures, in which case MCS is well defined. For MCS, we have  $\nu_c \equiv 1, \forall c$ , and therefore the corresponding measurement condition is
$$
m _ {c} \geq c _ {\epsilon / 2} \cdot \alpha^ {- 1} \cdot \lambda_ {c} (\widetilde {\mathbb {F}}) \cdot \log (2 d \max  \{n, 1 \} / \delta), \quad c = 1, \dots , C,
$$
where  $\lambda_c(\widetilde{\mathbb{F}}) = \operatorname{ess}\sup_{\theta \sim \rho_c}K_c(\widetilde{\mathbb{F}})(\theta)\geq \kappa_c(\widetilde{\mathbb{F}})$ . Comparing with (4.3), we conclude that the benefit of CS over MCS in terms of sample complexity corresponds to the difference between the supremum of the Christoffel function (i.e.,  $\lambda_c(\widetilde{\mathbb{F}})$ ) and its average (i.e.,  $\kappa_c(\widetilde{\mathbb{F}})$ ). Thus, if  $K_{c}$  has sharp peaks - as it does, for instance, in Fig. 2 - one expects a significant improvement from CS over MCS.
4.3 Generalization error bound and noisy data
Thus far, we have derived CS and shown parts (i) and (iii) of Theorem 2.7. In this section we establish the generalization bound in part (ii). For additional generality, we now consider noisy samples
$$
y _ {i c} = L _ {c} \left(\Theta_ {i c}\right) \left(f ^ {*}\right) + e _ {i c} \in \mathbb {Y} _ {c}, \quad i = 1, \dots , m _ {c} c = 1, \dots , C, \tag {4.4}
$$
where the  $e_{ic}$  represent measurement noise (see Remark E.6 for some further discussion on the noise term). We also consider inexact minimizers. Specifically, we say that  $\hat{f} \in \mathbb{F}$  is a  $\gamma$ -minimizer of (2.2) for some  $\gamma > 0$  if it yields a value of the objective function that is within  $\gamma$  of the minimum value. For example,  $\hat{f}$  may be the output of some training algorithm used to solve (2.2).
Theorem 4.8. Let  $0 < \epsilon, \delta, \gamma < 1$  and consider the setup of Section 2.1, except with noisy data (4.4). Suppose that (4.1) holds and also that  $\rho_c(D_c) < \infty$ ,  $\forall c$ . Then, for any  $f^* \in \mathbb{X}_0$  and  $\sigma \geq \| f^* \|_{\mathbb{X}}$ , the estimator  $\check{f} = \mathcal{T}_{\sigma}(\hat{f})$ , where  $\hat{f}$  is a  $\gamma$ -minimizer of (2.2), satisfies
$$
\mathbb {E} \| f ^ {*} - \check {f} \| _ {\mathbb {X}} ^ {2} \leq 3 \left(1 + \frac {4 \beta}{(1 - \epsilon) \alpha}\right) \inf _ {f \in \mathbb {F}} \| f ^ {*} - f \| _ {\mathbb {X}} ^ {2} + 4 \sigma^ {2} \delta + \frac {1 2}{(1 - \epsilon) \alpha} N + \frac {4}{(1 - \epsilon) \alpha} \gamma^ {2},
$$
where  $N = \sum_{c = 1}^{C}\frac{\rho_c(D_c)}{m_c}\sum_{i = 1}^{m_c}\| e_{ic}\|_{\mathbb{Y}_c}^2$
5 Conclusions, limitations and future work
We conclude by noting several limitations and areas for future work. First, in this work, we have striven for breadth – i.e., highlighting the efficacy of CS4ML across a diverse range of examples – rather than depth. Our aim is to show that the well-known ideas for active learning in standard regression (e.g., leverage score sampling) can be extended to a very general setting. We do not claim that CS is the best possible method for each example, and as such, our experimental results only compare against (inactive) MCS. In each application considered, there are other domain-specific strategies that are known to outperform MCS. See [5, 6, 22, 55, 74, 104, 111] and references therein for Fourier imaging, and [10, 31, 52, 51, 86, 113] in the case of PINNs. CS is a general framework, and is arguably more theoretically grounded and less heuristic than some such methods. Nonetheless, further investigation is needed to ascertain which method is best in each setting. In a similar vein, while Figs. 1–3 show significant performance gains from our method, the extent of such gains depends heavily on the problem. In Appendices B.6 and D.6 we discuss cases where the gains are far more marginal. In the PINNs example in particular, additional studies are needed to see if CAS leads to benefits across a wider spectrum of PDEs. Moreover, there is the intriguing possibility of using CS as the starting point for more advanced active learning schemes – for example, by generalizing the linear-sample sparsification techniques of [32] or the ‘boosting’ techniques of [58].
Second, as noted previously, a limitation of our theoretical analysis is the log-linear scaling with respect to the number of subspaces  $d$  (see Theorem 2.7, part (iii)). While this can be overcome in cases such as sparse regression (see Appendix A.4), we expect a more refined theoretical analysis may be able to tackle this problem in the general setting. Another limitation of our analysis is that the sample complexity bound in Theorem 4.2 involves  $K_{c}$  evaluated over the subspace cover  $\widetilde{\mathbb{F}}$ , rather than simply  $\mathbb{F} - \mathbb{F}$  itself. We anticipate this can also be improved through a more sophisticated argument. This would help close the theoretical gap in the generative models example considered in this paper. Another interesting theoretical direction involves reducing the sample complexity from log-linear to linear (when  $\mathbb{F}$  is a linear subspace), by extending, for example, [32].
Finally, we reiterate that our framework and theoretical analysis are both very general. Consequently, there are many other potential problems to which we can apply this work. As noted, the main practical hurdle in applying this framework to other problems is to determine how to sample from the optimal measure (2.7), or some computationally tractable surrogate. Some other problems of interest include low-rank matrix or tensor recovery, as mentioned briefly in Remark 4.5, sparse regression using random feature models, as was recently developed in [64], active learning for single neuron models, as developed in [50], and operator learning [17, 73]. These are interesting avenues for future work.
BA acknowledges the support of the Natural Sciences and Engineering Research Council of Canada of Canada (NSERC) through grant RGPIN-2021-611675. ND acknowledges the support of Florida State University through the CRC 2022-2023 FYAP grant program.