SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ben Adcock
ben_adcock@sfu.ca
Juan M. Cardenas
juan_manuel_cardenas@sfu.ca
Nick Dexter
Department of Scientific Computing
nick.dexter@fsu.edu
We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data acquired along continuous curves, and, multimodal data (i.e., combinations of different types of measurements). Our framework considers random sampling according to a finite number of sampling measures and arbitrary nonlinear approximation spaces (model classes). We introduce the concept of generalized Christoffel functions and show how these can be used to optimize the sampling measures. We prove that this leads to near-optimal sample complexity in various important cases. This paper focuses on applications in scientific computing, where active learning is often desirable, since it is usually expensive to generate data. We demonstrate the efficacy of our framework for gradient-augmented learning with polynomials, Magnetic Resonance Imaging (MRI) using generative models and adaptive sampling for solving PDEs using Physics-Informed Neural Networks (PINNs).
The standard regression problem in machine learning involves learning an approximation to a function  $f^{*}:D\subseteq \mathbb{R}^{d}\to \mathbb{R}$  from training data  $\{(\theta_i,f^* (\theta_i))\}_{i = 1}^m\subset D\times \mathbb{R}$ . The approximation is sought in a set of functions  $\mathbb{F}$ , typically termed a model class, hypothesis set or approximation space, and is often computed by minimizing the empirical error (or risk) over the training set, i.e.,
$$
\hat {f} \in \operatorname * {a r g m i n} _ {f \in \mathbb {F}} \frac {1}{m} \sum_ {i = 1} ^ {m} | f ^ {*} (\theta_ {i}) - f (\theta_ {i}) | ^ {2}.
$$
In this paper, we develop a generalization of this problem. This allows for general types of data (i.e., not just discrete function samples), including multimodal data, and random sampling from arbitrary distributions. In particular, this framework facilitates active learning by allowing one to optimize the sampling distributions to obtain near-best generalization from as few samples as possible.
1.1 Motivations
Typically, the sample points  $\theta_{i}$  in the above problem are drawn i.i.d. from some fixed distribution. Yet, in many applications of machine learning there is substantial freedom to choose the sample points in a more judicious manner to increase the generalization performance of the learned approximation. These applications are often highly data starved, thus making a good choice of sample points extremely valuable. Notably, many recent applications of machine learning in scientific computing often have these characteristics. An incomplete list of such applications include: Deep Learning (DL) for Partial Differential Equations (PDEs) via so-called Physics-Informed Neural Networks (PINNs) [39, 61, 83, 105, 112]; deep learning for parametric PDEs and operator learning [17, 49, 53, 66, 73, 75, 118]; machine learning for computational imaging [5, 79, 84, 88, 100, 107, 110]; and machine learning for discovering dynamics of physical systems [24, 25, 109].
In many such applications, the training data does not consist of pointwise samples of a target function. For example, in computational imaging, the samples are obtained through an integral transform such as the Fourier transform in Magnetic Resonance Imaging (MRI) or the Radon transform in X-ray Computed Tomography (CT) [5, 13, 21, 45]. In other applications, each sample may correspond to the value of  $f^{*}$  (or some integral transform thereof) along a continuous curve. Applications where this occurs include: seismic imaging, where physical sensors record continuously in time but at discrete spatial locations; discovering governing equations of physical systems, where each sample may be a full solution trajectory [109]; MRI, since the MR scanner samples along a sequence of continuous curves in  $k$ -space [80, 89]; and many more. In other applications, each sample may be a vector. For instance, in gradient-augmented learning problems – with applications to PINNs for PDEs [54, 120] and learning parametric PDEs in Uncertainty Quantification (UQ) [9, 56, 78, 82, 103, 98, 99], as well as various other DL settings [36] – each sample is a vector in  $\mathbb{R}^{d+1}$  containing the values of  $f$  and its gradient at  $\theta_{i}$ . In other applications, each sample may be an element of a function space. This occurs in parametric PDEs and operator learning, where each sample is the solution of a PDE (i.e., an element of a Hilbert space) corresponding to the given parameter value. Finally, many problems involve multimodal data, where one has  $C > 1$  different types of training data. This situation occurs in various applications, including: multi-sensor imaging systems [33] such as parallel MRI (which is ubiquitous in modern clinical practice) [5, 89]; PINNs for PDEs, where the data arises from the domain, the initial conditions and the boundary conditions; and multifidelity modelling [44, 102].
In this paper, we introduce a general framework for active learning in which
1.  $f^{*}$  need not be a scalar-valued function, but simply an element of a Hilbert space  $\mathbb{X}$ ;
2. the data arises from arbitrary linear operators, which may be scalar- or vector-valued;
3. the data may be multimodal, arising from  $C \geq 1$  of different types of linear operators;
4. the approximation space  $\mathbb{F}$  may be an arbitrary linear or nonlinear space;
5. sampling is random according to arbitrary probability measures;
6. the sample complexity, i.e., the relation between  $\mathbb{F}$  and the number of samples required to attain a near-best generalization error, is given explicitly in terms of the so-called generalized Christoffel functions of  $\mathbb{F}$ ;
7. using this explicit expression, the sampling measures can be optimized, leading to essentially optimal sample complexity in various cases.
This framework, Christoffel Sampling for Machine Learning (CS4ML), is very general. Indeed, we are unaware of any other active learning strategy that can simultaneously address such a broad class of problems. It is inspired by previous work on function regression in linear spaces [2, 34, 57, 60] and is also a generalization of leverage score sampling [11, 30, 32, 41, 46, 50, 85] for active learning. See Appendix A for further discussion. It generalizes these approaches by (a) considering abstract objects in Hilbert spaces, (b) allowing for arbitrary linear sampling operators, as opposed to just function samples, (c) allowing for multimodal data, and (d) considering general linear or nonlinear approximation spaces. We demonstrate its practical efficacy on a diverse set of test problems. These are: (i) Polynomial regression with gradient-augmented data, (ii) MRI reconstruction using generative models and (iii) Adaptive sampling for solving PDEs with PINNs. Our solution to (iii) introduces a general adaptive sampling strategy, termed Christoffel Adaptive Sampling, that can be applied to any DL-based regression problem (i.e., not simply PINNs for PDEs).
2 The CS4ML framework and main results
2.1 Main definitions
Let  $(\Omega, \mathcal{F}, \mathbb{P})$  be a probability space, which is the underlying probability space of the problem. In particular, our various results hold in probability with respect to the probability measure  $\mathbb{P}$ . Let  $\mathbb{X}$  be a separable Hilbert space and  $\mathbb{X}_0 \subseteq \mathbb{X}$  be a normed vector subspace of  $\mathbb{X}$ , termed the object space. Our goal is to learn an element (the object)  $f^* \in \mathbb{X}_0$  from training data. We consider a multi-modal measurement model in which  $C \geq 1$  different processes generate this data. For each  $c = 1, \dots, C$ , we assume there is a measure space  $(D_c, \mathcal{A}_c, \rho_c)$ , termed the measurement domain, which parametrizes the possible measurements. We will often consider the case where this is a probability space, but this is not strictly necessary for the moment. Next, we assume there are semi-inner product spaces  $\mathbb{Y}_c$ ,  $c = 1, \dots, C$ , termed measurement spaces, to which the measurements belong, and mappings
$$
L _ {c}: D _ {c} \to \mathcal {B} (\mathbb {X} _ {0}, \mathbb {Y} _ {c}), \quad c = 1, \dots , C,
$$
where  $\mathcal{B}(\mathbb{X}_0,\mathbb{Y}_c)$  denotes the space of bounded linear operators  $\mathbb{X}_0\to \mathbb{Y}_c$ . We refer to  $L_{c}$  as sampling operator for the  $c$ th measurement process. Note that we consider the sampling operators  $L_{c}$  as fixed and specified by the problem under consideration. However, we make the following assumption, which states that we have the flexibility to query each sampling operator at an arbitrary point in the measurement domain.
Assumption 2.1 (Active learning) For each  $c$ , we may query  $L_{c}(\theta)(f^{*})$  at any  $\theta \in D_{c}$ .
This assumption is a direct extension of that made in standard active learning (see Example 2.3 below). It holds in all our main examples. Given this assumption, our approach is to selecting sample points is to draw them randomly according to certain sampling measures  $\mu_1,\ldots ,\mu_C$  (defined on  $D_{1},\ldots ,D_{C}$ , respectively), which can then be optimized to ensure as good generalization as possible. To this end, we also make the following assumption.
Assumption 2.2 (Sampling measures) Each sampling measure  $\mu_{c}$  is a probability measure on  $(D_c, \mathcal{A}_c)$  that is absolutely continuous with respect to  $\rho_{c}$  and its Radon-Nikodym derivative is positive almost everywhere. In other words,  $\mathrm{d}\mu_{c}(\theta) = \nu_{c}(\theta) \mathrm{d}\rho_{c}(\theta)$  for some measurable function  $\nu_{c}: D_{c} \to \mathbb{R}$  that positive almost everywhere and satisfies  $\int_{D_c} \nu_{c}(\theta) \mathrm{d}\rho_{c}(\theta) = 1$ .
We now define the training data. Let  $m_{1},\ldots ,m_{C}$  be the (not necessarily equal) measurement numbers, i.e.,  $m_{c}$  is the number of measurements from the  $c$ th measurement process. Let  $\Theta_c:\Omega \to D_c$  be independent  $D_{c}$ -valued random variables, where  $\Theta_c\sim \mu_c$  for each  $c$ . For each  $c = 1,\dots ,C$ , let  $\Theta_{ic},i = 1,\dots ,m_c$ , be independent realizations of  $\Theta_c$ . Then the training data is
$$
\left\{\left(\Theta_ {i c}, y _ {i c}\right) \right\} _ {i, c = 1} ^ {m _ {c}, C}, \quad \text {w h e r e} y _ {i c} = L _ {c} \left(\Theta_ {i c}\right) \left(f ^ {*}\right) \in \mathbb {Y} _ {c}, i = 1, \dots , m _ {c} c = 1, \dots , C. \tag {2.1}
$$
Later, in Section 4.3 we also allow for noisy measurements. Note that we consider the agnostic learning setting, where  $f^{*} \notin \mathbb{F}$  and the noise can be adversarial, as this setting is most appropriate to the various applications considered (see also [50]).
Example 2.3 (Active learning in standard regression) The above framework extends the standard active learning problem in regression. In the classic regression problem,  $D \subseteq \mathbb{R}^d$  is a domain and  $\mathbb{X} = L_{\rho}^{2}(D)$  is the space of square-integrable functions  $f^{*}: D \to \mathbb{R}$  with respect to a measure  $\rho$ . Note that  $\rho$  is considered fixed – it is the measure with respect to which we measure the error. To embed this problem into the above framework, we let  $\mathbb{X}_0 = C(\overline{D})$  be the space of continuous functions on  $D$ ,  $C = 1$ , the measurement domain  $D_c = D$  be equal to the domain of the function,  $\rho_1 = \rho$  and the measurement space  $\mathbb{Y}_1 = \mathbb{R}$  (with the Euclidean inner product). We then define the sampling operator  $L_1(\theta)(f^*) = f^*(\theta)$  as the pointwise evaluation operator. In particular, for a measure  $\mu = \mu_1$  satisfying Assumption 2.2, the training data (2.1) is  $(\Theta_i, f^*(\Theta_i))$  with  $\Theta_i \sim_{\mathrm{i.i.d.}} \mu$ . Hence, the aim is to choose the measure  $\mu$  (or equivalently, its Radon-Nikodym derivative  $\nu$ ) to ensure as good generalization as possible.
Next, we let  $\mathbb{F} \subseteq \mathbb{X}_0$  be a subset within which we seek to learn  $f^*$ . We term  $\mathbb{F}$  the approximation space. Note this could be a linear space such as a space of algebraic or trigonometric polynomials, or a nonlinear space such as the space of sparse Fourier functions, a space of functions with sparse
representations in a multiscale system such as wavelets, the space corresponding to a single neuron model, or a space of deep neural networks. We discuss a number of examples later. Given  $\mathbb{F}$  and  $f^{*} \in \mathbb{X}_{0}$ , in this work we consider an approximation  $\hat{f} \in \mathbb{F}$  defined via the empirical least-squares fit
$$
\hat {f} \in \operatorname * {a r g m i n} _ {f \in \mathbb {F}} \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu_ {c} \left(\Theta_ {i c}\right)} \| y _ {i c} - L _ {c} \left(\Theta_ {i c}\right) (f) \| _ {\mathbb {Y} _ {c}} ^ {2}. \tag {2.2}
$$
However, regardless of the method employed, one cannot expect to learn  $f^{*}$  from (2.1) without an assumption on the sampling operators. Indeed, consider the case  $L_{c} \equiv 0$ ,  $\forall c$ . The following assumption states that the sampling operators should be sufficiently 'rich' to approximately preserve the  $\mathbb{X}$ -norm. As we see later, this is a natural assumption, which holds in many cases of interest.
Assumption 2.4 (Nondegeneracy of the sampling operators) The mappings  $L_{c}$  are such that the maps  $D_{c} \to \mathbb{Y}_{c}, \theta \mapsto L_{c}(\theta)(f)$  are measurable for every  $f \in \mathbb{X}_0$ . Moreover, the functions  $D_{c} \to \mathbb{C}, \theta \mapsto \|L_{c}(\theta)(f)\|_{\mathbb{Y}_{c}}^{2}$  are integrable and satisfy, for constants  $0 < \alpha \leq \beta < \infty$ ,
$$
\alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \sum_ {c = 1} ^ {C} \int_ {D _ {c}} \| L _ {c} (\theta) (f) \| _ {\mathbb {Y} _ {c}} ^ {2} \mathrm {d} \rho_ {c} (\theta) \leq \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {X} _ {0}. \tag {2.3}
$$
In order to successfully learn  $f^{*}$  from the samples (2.1), we need to (approximately) preserve nondegeneracy when the integrals in (2.3) are replaced by discrete sums. Notice that
$$
\alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \mathbb {E} \left(\sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (f) \| _ {\mathbb {Y} _ {c}} ^ {2}\right) \leq \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {X} _ {0}, \tag {2.4}
$$
where  $\mathbb{E}$  denotes the expectation with respect to all the random variables  $(\Theta_{ic})_{i,c}$ . Our subsequent analysis involves deriving conditions under which this holds with high probability. We shall say that empirical nondegeneracy holds with constant  $0 < \epsilon < 1$  for a draw  $(\Theta_{ic})_{i,c}$  if
$$
(1 - \epsilon) \alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (f) \| _ {\mathbb {Y} _ {c}} ^ {2} \leq (1 + \epsilon) \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {F} - \mathbb {F}, \tag {2.5}
$$
where  $\mathbb{F} - \mathbb{F} = \{f_1 - f_2 : f_1, f_2 \in \mathbb{F}\}$ . See Appendix A for some background on this definition.
Finally, we need one further assumption. This assumption essentially says that the action of the sampling operator  $L_{c}$  on  $\mathbb{F}$  is nontrivial, in the sense that, for almost all  $\theta \in D_{c}$ , there exists a  $f \in \mathbb{F}$  that has a nonzero measurement  $L_{c}(\theta)(f)$ . This assumption is reasonable, since without it, there is little hope to fit the measurements with an approximation from the space  $\mathbb{F}$ .
Assumption 2.5 (Nondegeneracy of  $\mathbb{F}$  with respect to  $L_{c}$ ) For each  $c = 1, \ldots, C$  and almost every  $\theta_{c} \in D_{c}$ , there exists an element  $f_{c} \in \mathbb{F}$  such that  $L_{c}(\theta_{c})(f_{c}) \neq 0$ .
2.2 Summary of main results
The goal of this work is to derive sampling measures that ensure a quasi-optimal generalization bound from as little training data as possible. These measures are given in terms of the following function.
Definition 2.6 (Generalized Christoffel function). Let  $\mathbb{X}_0$  be normed vector space,  $\mathbb{Y}$  be a semiinner product space,  $(D,\mathcal{A},\rho)$  be a measure space,  $L:D\to \mathcal{B}(\mathbb{X}_0,\mathbb{Y})$  be such that the function  $D\rightarrow \mathbb{Y},\theta \mapsto L(\theta)(f)$  is measurable for every  $f\in \mathbb{X}_0$  and  $\mathbb{F}\subseteq \mathbb{X}_0$ $\mathbb{F}\neq \{0\}$ . The Generalized Christoffel function of  $\mathbb{F}$  with respect to  $L$  is the function
$$
K (\mathbb {F}) (\theta) = \sup  \{\| L (\theta) (f) \| _ {\mathbb {Y}} ^ {2} / \| f \| _ {\mathbb {X}} ^ {2}: f \in \mathbb {F}, f \neq 0 \}, \quad \theta \in D.
$$
If  $\mathbb{F} = \{0\}$ , then we set  $K(\mathbb{F})(\theta) = 0$ . Further, we also define  $\kappa(\mathbb{F}) = \int_{D} K(\mathbb{F})(\theta) \, \mathrm{d}\rho(\theta)$ .
In the standard regression problem (see Example 2.3), Definition 2.6 reduces to
$$
K (\mathbb {F}) (\theta) = \sup  \left\{\left| f (\theta) \right| ^ {2} / \| f \| _ {L _ {\rho} ^ {2} (D)} ^ {2}: f \in \mathbb {F}, f \neq 0 \right\}, \quad \theta \in D. \tag {2.6}
$$
This is a well-known object, which is often referred to as the Christoffel function when  $\mathbb{F}$  is a linear subspace. It is also equivalent (up to a scaling) to the leverage score function. See Appendix A.2 for further discussion. Definition 2.6 extends these notions from linear subspaces to nonlinear spaces and from pointwise samples to arbitrary sampling operators  $L$ .
To state our main result, we also need the following. Given  $\sigma > 0$  we define the shrinkage operator  $\mathcal{T}_{\sigma}:\mathbb{X}\to \mathbb{X}$  by  $\mathcal{T}_{\sigma}(f) = \min \left\{1,\sigma /\| f\|_{\mathbb{X}}\right\} f,\forall f\in \mathbb{X}$ .
Theorem 2.7. Consider the setup of Section 2.1, where  $\mathbb{F} - \mathbb{F} = \cup_{j=1}^{d}\mathbb{F}_{j}$  is a union of  $d \in \mathbb{N}$  subspaces of dimension at most  $n \in \mathbb{N}$ . Let  $K_{c}, c = 1,\dots,C$ , be as in Definition 2.6 for  $(D,\mathcal{A},\rho) = (D_{c},\mathcal{A}_{c},\rho_{c})$  and  $L = L_{c}$ . Suppose that the  $K_{c}(\mathbb{F} - \mathbb{F})$  are integrable, define
$$
\mathrm {d} \mu_ {c} ^ {\star} (\theta) = K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) \mathrm {d} \rho_ {c} (\theta) / \kappa_ {c} (\mathbb {F} - \mathbb {F}), \quad c = 1, \dots , C, \tag {2.7}
$$
and suppose that
$$
m _ {c} \asymp \alpha^ {- 1} \cdot \kappa_ {c} (\mathbb {F} - \mathbb {F}) \cdot \log (2 n d / \delta), \quad c = 1, \dots , C \tag {2.8}
$$
for some  $0 < \delta < 1$ . Then the following hold.
(i) Empirical nondegeneracy (2.5) holds with  $\epsilon = 1 / 2$  with probability at least  $1 - \delta$ .
(ii) For any  $f^{*}\in \mathbb{X}_{0},\| f^{*}\|_{\mathbb{X}}\leq 1$  , the estimator  $\tilde{f} = \mathcal{T}_1(\hat{f})$  , where  $\hat{f}$  is as in (2.2), satisfies
$$
\mathbb {E} \| f ^ {*} - \check {f} \| _ {\mathbb {X}} ^ {2} \lesssim (\beta / \alpha) \cdot \inf  _ {f \in \mathbb {F}} \| f ^ {*} - f \| _ {\mathbb {X}} ^ {2} + \delta . \tag {2.9}
$$
(iii) The total number of samples  $m = m_{1} + \dots + m_{C}$  is at most log-linear in  $nd$ , namely,
$$
m \lesssim (\beta / \alpha) \cdot n d \cdot \log (2 n d / \delta). \tag {2.10}
$$
The choice (2.7) is a particular type of importance sampling, which we term Christoffel Sampling (CS). In particular, for the standard regression problem (see Example 2.3) the resulting measure is
$$
\mathrm {d} \mu^ {\star} (y) \propto K (\mathbb {F} - \mathbb {F}) (y) \mathrm {d} \rho (y), \tag {2.11}
$$
where  $K(\mathbb{F} - \mathbb{F})(y)$  is as in (2.6) with  $\mathbb{F}$  replaced by  $\mathbb{F} - \mathbb{F}$ . As we discuss in Appendix A.2, this is equivalent to leverage score sampling for the standard regression problem (see, e.g., [2, 12, 34, 32, 46, 60]). Therefore, Theorem 2.7 can be considered a generalization of leverage score sampling from standard regression to significantly more general types of linear, multimodal measurements. Specifically, it considers the general setting of arbitrary Hilbert spaces  $\mathbb{X}$ , nonlinear approximation spaces  $\mathbb{F}$ , and  $C > 1$  arbitrary, linear sampling operators  $L_{c}$  taking values in arbitrary semi-inner product spaces  $\mathbb{Y}_{c}$ . To the best of our knowledge this generalization is new. We remark, however, that Theorem 2.7 is also related to the recent work [43]. See Appendix A.3 for further discussion.
The sampling measure (2.7) is 'optimal' in the sense that it minimizes a sufficient condition over all possible sampling measures  $\mu_c$  (see Lemma 4.6). Doing so leads to CS and Theorem 2.7. The measurement condition (2.10) states that CS leads to a desirable sample complexity bound (2.10) that is at worse log-linear in  $nd$ . In particular, when  $d = 1$  (i.e.,  $\mathbb{F}$  is a linear space) the sample complexity is near-optimal (and in this case, one also has  $\mathbb{F} - \mathbb{F} = \mathbb{F}$ ). In general, the bound (2.10) is near-optimal in terms of  $n$  when  $d$  is small (and independent of  $n$ ). Unfortunately,  $d$  can be large in important cases such as sparse regression. In this case, however, the linear dependence on  $d$  in (2.10) – which follows from (2.8) via the crude estimate  $\sum_{c=1}^{C} \kappa_c (\mathbb{F} - \mathbb{F}) \leq \beta nd$  – can be lessened by using the specific structure of the approximation space. See Appendix A.4.
Theorem 2.7 is a simplification and combination of our several main results. In Theorem 4.2 we consider more general approximation spaces, which need not be unions of finite-dimensional subspaces. See also Remark 4.5. Finally, we note that  $\tilde{f}$  only differs from  $\hat{f}$  by a shrinkage operator, which is a technical step needed to bound the error in expectation. See Section 4.3. An 'in probability' bound can be obtained without this additional complication, albeit with less succinct right-hand side.
3 Examples and numerical experiments
Theorem 2.7 shows that CS can be a near-optimal active learning strategy. We now show its efficacy via a series of examples. These examples also highlight how the generality of the framework allows it to tackle a wide range of problems. We consider cases where the theory applies directly and others
Figure 1: CS for gradient-augmented polynomial regression. Plots of the average regression error versus  $m$  for MCS and CS for gradient-augmented polynomial regression of the function  $f^{*}(\theta) = \exp (-(\theta_{1} + .\dots +\theta_{d}) / (2d))$  for  $d = 2,4,8$  (left to right). In both cases, we use the scaling  $m = \lceil \max \{n,n\log (n)\} /(d + 1)\rceil$ . In this and other figures, the solid lines show the mean value and the shaded region indicates one standard deviation. See Section B.5 for further details.
where it does not. In all cases, we present performance gains over inactive learning, i.e., Monte Carlo Sampling (MCS) from the underlying probability measure. Another matter we address is how to sample from the measures (2.7) in practice, this being very much dependent on the problem considered. Full experimental details for each application can be found in Appendices B-D.
3.1 Polynomial regression with gradient-augmented data