Title: CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: active learning

regression

arbitrary data

leverage scores

Christoffel functions

generative models

Magnetic Resonance Imaging (MRI)

Physics-Informed Neural Networks (PINNs)
Abstract: We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data acquired along continuous curves, and, multimodal data (i.e., combinations of different types of measurements).  Our framework considers random sampling according to a finite number of sampling measures and arbitrary nonlinear approximation spaces (model classes). We introduce the concept of \textit{generalized Christoffel functions} and show how these can be used to optimize the sampling measures. We prove that this leads to near-optimal sample complexity in various important cases. This paper focuses on applications in scientific computing, where active learning is often desirable, since it is usually expensive to generate data. We demonstrate the efficacy of our framework for gradient-augmented learning with polynomials, Magnetic Resonance Imaging (MRI) using generative models and adaptive sampling for solving PDEs using Physics-Informed Neural Networks (PINNs).

CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions

Abstract

We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data acquired along continuous curves, and, multimodal data (i.e., combinations of different types of measurements). Our framework considers random sampling according to a finite number of sampling measures and arbitrary nonlinear approximation spaces (model classes). We introduce the concept of generalized Christoffel functions and show how these can be used to optimize the sampling measures. We prove that this leads to near-optimal sample complexity in various important cases. This paper focuses on applications in scientific computing, where active learning is often desirable, since it is usually expensive to generate data. We demonstrate the efficacy of our framework for gradient-augmented learning with polynomials, Magnetic Resonance Imaging (MRI) using generative models and adaptive sampling for solving PDEs using Physics-Informed Neural Networks (PINNs).

1 Introduction

The standard regression problem in machine learning involves learning an approximation to a function f ∗ : D ⊆ R d → R from training data { ( θ i , f ∗ ( θ i )) } i m =1 ⊂ D × R . The approximation is sought in a set of functions F , typically termed a model class , hypothesis set or approximation space , and is often computed by minimizing the empirical error (or risk) over the training set, i.e.,

In this paper, we develop a generalization of this problem. This allows for general types of data (i.e., not just discrete function samples), including multimodal data, and random sampling from arbitrary distributions. In particular, this framework facilitates active learning by allowing one to optimize the sampling distributions to obtain near-best generalization from as few samples as possible.

1.1 Motivations

Typically, the sample points θ i in the above problem are drawn i.i.d. from some fixed distribution. Yet, in many applications of machine learning there is substantial freedom to choose the sample points in a more judicious manner to increase the generalization performance of the learned approximation. These applications are often highly data starved , thus making a good choice of sample points extremely valuable. Notably, many recent applications of machine learning in scientific computing often have these characteristics. An incomplete list of such applications include: Deep Learning (DL) for Partial Differential Equations (PDEs) via so-called Physics-Informed Neural Networks (PINNs) [39, 61, 83, 105, 112]; deep learning for parametric PDEs and operator learning [17, 49, 53, 66, 73, 75, 118]; machine learning for computational imaging [5, 79, 84, 88, 100, 107, 110]; and machine learning for discovering dynamics of physical systems [24, 25, 109].

In many such applications, the training data does not consist of pointwise samples of a target function. For example, in computational imaging, the samples are obtained through an integral transform such as the Fourier transform in Magnetic Resonance Imaging (MRI) or the Radon transform in X-ray Computed Tomography (CT) [5, 13, 21, 45]. In other applications, each sample may correspond to the value of f ∗ (or some integral transform thereof) along a continuous curve. Applications where this occurs include: seismic imaging, where physical sensors record continuously in time but at discrete spatial locations; discovering governing equations of physical systems, where each sample may be a full solution trajectory [109]; MRI, since the MR scanner samples along a sequence of continuous curves in k -space [80, 89]; and many more. In other applications, each sample may be a vector. For instance, in gradient-augmented learning problems - with applications to PINNs for PDEs [54, 120] and learning parametric PDEs in Uncertainty Quantification (UQ) [9, 56, 78, 82, 103, 98, 99], as well as various other DL settings [36] - each sample is a vector in R d +1 containing the values of f and its gradient at θ i . In other applications, each sample may be an element of a function space. This occurs in parametric PDEs and operator learning, where each sample is the solution of a PDE (i.e., an element of a Hilbert space) corresponding to the given parameter value. Finally, many problems involve multimodal data , where one has C > 1 different types of training data. This situation occurs in various applications, including: multi-sensor imaging systems [33] such as parallel MRI (which is ubiquitous in modern clinical practice) [5, 89]; PINNs for PDEs, where the data arises from the domain, the initial conditions and the boundary conditions; and multifidelity modelling [44, 102].

1.2 Contributions

In this paper, we introduce a general framework for active learning in which

f ∗ need not be a scalar-valued function, but simply an element of a Hilbert space X ;

the data arises from arbitrary linear operators, which may be scalar- or vector-valued;

the data may be multimodal, arising from C ≥ 1 of different types of linear operators;

the approximation space F may be an arbitrary linear or nonlinear space;

sampling is random according to arbitrary probability measures;

the sample complexity , i.e., the relation between F and the number of samples required to attain a near-best generalization error, is given explicitly in terms of the so-called generalized Christoffel functions of F ;

using this explicit expression, the sampling measures can be optimized, leading to essentially optimal sample complexity in various cases.

This framework, Christoffel Sampling for Machine Learning (CS4ML) , is very general. Indeed, we are unaware of any other active learning strategy that can simultaneously address such a broad class of problems. It is inspired by previous work on function regression in linear spaces [2, 34, 57, 60] and is also a generalization of leverage score sampling [11, 30, 32, 41, 46, 50, 85] for active learning. See Appendix A for further discussion. It generalizes these approaches by (a) considering abstract objects in Hilbert spaces, (b) allowing for arbitrary linear sampling operators, as opposed to just function samples, (c) allowing for multimodal data, and (d) considering general linear or nonlinear approximation spaces. We demonstrate its practical efficacy on a diverse set of test problems. These are: (i) Polynomial regression with gradient-augmented data , (ii) MRI reconstruction using generative models and (iii) Adaptive sampling for solving PDEs with PINNs . Our solution to (iii) introduces a general adaptive sampling strategy, termed Christoffel Adaptive Sampling , that can be applied to any DL-based regression problem (i.e., not simply PINNs for PDEs).

2 The CS4ML framework and main results

2.1 Main definitions

Let (Ω , F , P ) be a probability space, which is the underlying probability space of the problem. In particular, our various results hold in probability with respect to the probability measure P . Let X be a separable Hilbert space and X 0 ⊆ X be a normed vector subspace of X , termed the object space . Our goal is to learn an element (the object) f ∗ ∈ X 0 from training data. We consider a multi-modal measurement model in which C ≥ 1 different processes generate this data. For each c = 1 , . . . , C , we assume there is a measure space ( D c , A c , ρ c ) , termed the measurement domain , which parametrizes the possible measurements. We will often consider the case where this is a probability space, but this is not strictly necessary for the moment. Next, we assume there are semi-inner product spaces Y c , c = 1 , . . . , C , termed measurement spaces , to which the measurements belong, and mappings

where B ( X 0 , Y c ) denotes the space of bounded linear operators X 0 → Y c . We refer to L c as sampling operator for the c th measurement process. Note that we consider the sampling operators L c as fixed and specified by the problem under consideration. However, we make the following assumption, which states that we have the flexibility to query each sampling operator at an arbitrary point in the measurement domain.

Assumption 2.1 (Active learning) For each c , we may query L c ( θ )( f ∗ ) at any θ ∈ D c .

This assumption is a direct extension of that made in standard active learning (see Example 2.3 below). It holds in all our main examples. Given this assumption, our approach is to selecting sample points is to draw them randomly according to certain sampling measures µ 1 , . . . , µ C (defined on D 1 , . . . , D C , respectively), which can then be optimized to ensure as good generalization as possible. To this end, we also make the following assumption.

Assumption 2.2 (Sampling measures) Each sampling measure µ c is a probability measure on ( D c , A c ) that is absolutely continuous with respect to ρ c and its Radon-Nikodym derivative is positive almost everywhere. In other words, d µ c ( θ ) = ν c ( θ ) d ρ c ( θ ) for some measurable function ν c : D c → R that positive almost everywhere and satisfies ∫ D c ν c ( θ ) d ρ c ( θ ) = 1 .

We now define the training data. Let m 1 , . . . , m C be the (not necessarily equal) measurement numbers, i.e., m c is the number of measurements from the c th measurement process. Let Θ c : Ω → D c be independent D c -valued random variables, where Θ c ∼ µ c for each c . For each c = 1 , . . . , C , let Θ ic , i = 1 , . . . , m c , be independent realizations of Θ c . Then the training data is

Later, in Section 4.3 we also allow for noisy measurements. Note that we consider the agnostic learning setting, where f ∗ / ∈ F and the noise can be adversarial, as this setting is most appropriate to the various applications considered (see also [50]).

Example 2.3 (Active learning in standard regression) The above framework extends the standard active learning problem in regression. In the classic regression problem, D ⊆ R d is a domain and X = L 2 ρ ( D ) is the space of square-integrable functions f ∗ : D → R with respect to a measure ρ . Note that ρ is considered fixed - it is the measure with respect to which we measure the error. To embed this problem into the above framework, we let X 0 = C ( D ) be the space of continuous functions on D , C = 1 , the measurement domain D c = D be equal to the domain of the function, ρ 1 = ρ and the measurement space Y 1 = R (with the Euclidean inner product). We then define the sampling operator L 1 ( θ )( f ∗ ) = f ∗ ( θ ) as the pointwise evaluation operator. In particular, for a measure µ = µ 1 satisfying Assumption 2.2, the training data (2.1) is (Θ i , f ∗ (Θ i )) with Θ i ∼ i . i . d . µ . Hence, the aim is to choose the measure µ (or equivalently, its Radon-Nikodym derivative ν ) to ensure as good generalization as possible.

Next, we let F ⊆ X 0 be a subset within which we seek to learn f ∗ . We term F the approximation space . Note this could be a linear space such as a space of algebraic or trigonometric polynomials, or a nonlinear space such as the space of sparse Fourier functions, a space of functions with sparse

representations in a multiscale system such as wavelets, the space corresponding to a single neuron model, or a space of deep neural networks. We discuss a number of examples later. Given F and f ∗ ∈ X 0 , in this work we consider an approximation ˆ f ∈ F defined via the empirical least-squares fit

However, regardless of the method employed, one cannot expect to learn f ∗ from (2.1) without an assumption on the sampling operators. Indeed, consider the case L c ≡ 0 , ∀ c . The following assumption states that the sampling operators should be sufficiently 'rich' to approximately preserve the X -norm. As we see later, this is a natural assumption, which holds in many cases of interest.

Assumption 2.4 (Nondegeneracy of the sampling operators) The mappings L c are such that the maps D c → Y c , θ ↦→ L c ( θ )( f ) are measurable for every f ∈ X 0 . Moreover, the functions D c → C , θ ↦→∥ L c ( θ )( f ) ∥ 2 Y c are integrable and satisfy, for constants 0 < α ≤ β < ∞ ,

In order to successfully learn f ∗ from the samples (2.1), we need to (approximately) preserve nondegeneracy when the integrals in (2.3) are replaced by discrete sums. Notice that

where E denotes the expectation with respect to all the random variables (Θ ic ) i,c . Our subsequent analysis involves deriving conditions under which this holds with high probability. We shall say that empirical nondegeneracy holds with constant 0 < ϵ < 1 for a draw (Θ ic ) i,c if

where F -F = { f 1 -f 2 : f 1 , f 2 ∈ F } . See Appendix A for some background on this definition.

Finally, we need one further assumption. This assumption essentially says that the action of the sampling operator L c on F is nontrivial, in the sense that, for almost all θ ∈ D c , there exists a f ∈ F that has a nonzero measurement L c ( θ )( f ) . This assumption is reasonable, since without it, there is little hope to fit the measurements with an approximation from the space F .

̸

Assumption 2.5 (Nondegeneracy of F with respect to L c ) For each c = 1 , . . . , C and almost every θ c ∈ D c , there exists an element f c ∈ F such that L c ( θ c )( f c ) = 0 .

2.2 Summary of main results

The goal of this work is to derive sampling measures that ensure a quasi-optimal generalization bound from as little training data as possible. These measures are given in terms of the following function.

̸

Definition 2.6 (Generalized Christoffel function) . Let X 0 be normed vector space, Y be a semiinner product space, ( D, A , ρ ) be a measure space, L : D → B ( X 0 , Y ) be such that the function D → Y , θ ↦→ L ( θ )( f ) is measurable for every f ∈ X 0 and F ⊆ X 0 , F = { 0 } . The Generalized Christoffel function of F with respect to L is the function

̸

If F = { 0 } , then we set K ( F )( θ ) = 0 . Further, we also define κ ( F ) = ∫ D K ( F )( θ ) d ρ ( θ ) .

In the standard regression problem (see Example 2.3), Definition 2.6 reduces to

̸

This is a well-known object, which is often referred to as the Christoffel function when F is a linear subspace. It is also equivalent (up to a scaling) to the leverage score function . See Appendix A.2 for further discussion. Definition 2.6 extends these notions from linear subspaces to nonlinear spaces and from pointwise samples to arbitrary sampling operators L .

To state our main result, we also need the following. Given σ > 0 we define the shrinkage operator T σ : X → X by T σ ( f ) = min { 1 , σ/ ∥ f ∥ X } f , ∀ f ∈ X .

Theorem 2.7. Consider the setup of Section 2.1, where F -F = ∪ d j =1 F j is a union of d ∈ N subspaces of dimension at most n ∈ N . Let K c , c = 1 , . . . , C , be as in Definition 2.6 for ( D, A , ρ ) = ( D c , A c , ρ c ) and L = L c . Suppose that the K c ( F -F ) are integrable, define

and suppose that

for some 0 < δ < 1 . Then the following hold.

(i) Empirical nondegeneracy (2.5) holds with ϵ = 1 / 2 with probability at least 1 -δ .

(ii) For any f ∗ ∈ X 0 , ∥ f ∗ ∥ X ≤ 1 , the estimator ˇ f = T 1 ( ˆ f ) , where ˆ f is as in (2.2) , satisfies

(iii) The total number of samples m = m 1 + · · · + m C is at most log-linear in nd , namely,

The choice (2.7) is a particular type of importance sampling, which we term Christoffel Sampling (CS) . In particular, for the standard regression problem (see Example 2.3) the resulting measure is

where K ( F -F )( y ) is as in (2.6) with F replaced by F -F . As we discuss in Appendix A.2, this is equivalent to leverage score sampling for the standard regression problem (see, e.g., [2, 12, 34, 32, 46, 60]). Therefore, Theorem 2.7 can be considered a generalization of leverage score sampling from standard regression to significantly more general types of linear, multimodal measurements. Specifically, it considers the general setting of arbitrary Hilbert spaces X , nonlinear approximation spaces F , and C > 1 arbitrary, linear sampling operators L c taking values in arbitrary semi-inner product spaces Y c . To the best of our knowledge this generalization is new. We remark, however, that Theorem 2.7 is also related to the recent work [43]. See Appendix A.3 for further discussion.

The sampling measure (2.7) is 'optimal' in the sense that it minimizes a sufficient condition over all possible sampling measures µ c (see Lemma 4.6). Doing so leads to CS and Theorem 2.7. The measurement condition (2.10) states that CS leads to a desirable sample complexity bound (2.10) that is at worse log-linear in nd . In particular, when d = 1 (i.e., F is a linear space) the sample complexity is near-optimal (and in this case, one also has F -F = F ). In general, the bound (2.10) is near-optimal in terms of n when d is small (and independent of n ). Unfortunately, d can be large in important cases such as sparse regression. In this case, however, the linear dependence on d in (2.10) - which follows from (2.8) via the crude estimate ∑ C c =1 κ c ( F -F ) ≤ βnd - can be lessened by using the specific structure of the approximation space. See Appendix A.4.

Theorem 2.7 is a simplification and combination of our several main results. In Theorem 4.2 we consider more general approximation spaces, which need not be unions of finite-dimensional subspaces. See also Remark 4.5. Finally, we note that ˇ f only differs from ˆ f by a shrinkage operator, which is a technical step needed to bound the error in expectation. See Section 4.3. An 'in probability' bound can be obtained without this additional complication, albeit with less succinct right-hand side.

3 Examples and numerical experiments

Theorem 2.7 shows that CS can be a near-optimal active learning strategy. We now show its efficacy via a series of examples. These examples also highlight how the generality of the framework allows it to tackle a wide range of problems. We consider cases where the theory applies directly and others

一

MCS

MCS

MCS

100

CS

103

CS

100

CS

test error E(f)

a p

10-5

100

10~2

10-10

10-3

10-15

10-6

10-4

102

103

102

103

102

103

m

m

m

where it does not. In all cases, we present performance gains over inactive learning, i.e., Monte Carlo Sampling (MCS) from the underlying probability measure. Another matter we address is how to sample from the measures (2.7) in practice, this being very much dependent on the problem considered. Full experimental details for each application can be found in Appendices B-D.

3.1 Polynomial regression with gradient-augmented data

Many machine learning applications involve regressing a smooth, multivariate function using a model class consisting of algebraic polynomials. Often, the training data consists of function values. Yet, as noted in Section 1.1, there are many settings where one can acquire both function values and values of its gradient. In this case, the training data takes the form { ( θ i , f ( θ i ) , ∇ f ( θ i )) } i m =1 ⊂ R d × R d +1 . As we explain in Appendix B.1, this problem fits into our framework with C = 1 , where ρ is the standard Gaussian measure on R d , X = H 1 ρ ( R d ) is the Sobolev space of weighted square-integrable functions with weighted square-integrable first-order derivatives and Y = R d +1 with the Euclidean inner product. It therefore provides a first justification for allowing non-scalar valued sampling operators. Note that several important variations on this problem also fit into our general framework with C > 1 . See Appendix B.7.

We consider learning a function from such training data in a sequence of nested polynomial spaces F 1 ⊂ F 2 ⊂ · · · . These spaces are based on hyperbolic cross index sets, which are particularly useful in multivariate polynomial regression tasks [1, 37]. See Appendix B.2 for the formal definition.

In Fig. 1 we compare gradient-augmented polynomial regression with CS versus MCS from the Gaussian measure ρ . See Appendices B.3-B.5 for details on how we implement CS in this case. CS gives a dramatic improvement over MCS. CS is theoretically near-optimal in the sense of Theorem 2.7 - i.e., it provably yields a log-linear sample complexity - since the approximation spaces are linear subspaces in this example. On the other hand, MCS fails, with the error either not decreasing or diverging. The reason is the log-linear scaling, which is not sufficient to ensure a generalization error bound of the form (2.9) for MCS. As we demonstrate in Appendix B.6, MCS requires a much more severe scaling of m with n to ensure a generalization bound of this type.

3.2 MRI reconstruction using generative models

Reconstructing an image from measurements is a fundamental task in science, engineering and industry. In many applications - in particular, medical imaging modalities such as MRI - one wishes to reduce the number of measurements while still retaining image quality. As noted, techniques based on DL have recently led to significant breakthroughs in image recovery tasks. One promising approach involves using generative models [16, 20, 67]. First, a generative model is trained on a database of relevant images, e.g., brain images in the case of MRI. Then the image recovery problem is formulated as a regression problem, such as (2.2), where F is the range of the generative model. Note that in this problem, the training data consists of a finite set of frequencies and the corresponding values of the Fourier transform of the unknown image.

As we explain in Appendices C.1-C.3, this problem fits into our general framework. In Fig. 2 we demonstrate the efficacy of CS for Fourier imaging with generative models. In this example, the

30

40

50

60

70

80

90

5

10

15

20

25

30

35

40

45

0.25

0.5

0.75

1.0

1.25

1.5

1.75

2.0

15

20

25

30

35

40

45

10 -1

10 0

10 1

10 2

10 3

generative model was trained on a database of 3D MRI brain images (see Appendix C.5). This experiment simulates a 3D image reconstruction problem in MRI, where the measurements are samples of the Fourier transform of the unknown image taken along horizontal lines in k -space (a sampling strategy commonly known as phase encoding ). The active learning problem involves judiciously choosing m horizontal lines in k -space to enhance the generalization performance of the learning procedure. In Fig. 2, we compare the average Peak Signal-to-Noise Ratio (PSNR) versus frame (i.e., 2D image slice) number for CS versus MCS (i.e., uniform random sampling) for reconstructing an unknown image. We observe a significant improvement, especially in the challenging regime where the sampling percentage (the ratio of the number of measurements to the image size) is low.

This example lies close to our main theorem, but is not fully covered by it. See Appendix C.3. The space F (the range of a generative model) is not a finite union of subspaces. However, it is known that certain generative models (namely, those based on ReLU activation functions) are subsets of unions of subspaces of controllable size [16]. Further, we do not sample exactly from (2.7) in this case, but rather an empirical approximation to it (see Appendix C.4). Nevertheless, our experiments show a significant performance gain from CS in this case, despite it lying strictly outside of our theory.

This application justifies the presence of arbitrary (i.e., non-pointwise) sampling operators in our framework. It is another instance of non-scalar valued sampling operators, since each measurement is a vector of frequencies values along a horizontal line in k -space. Fig. 2 considers C = 1 sampling operators. However, as we explain Appendix C.7, the important extension of this setup to parallel MRI (which is standard in clinical practice) can be formulated in our framework with C > 1 .

3.3 Adaptive sampling for solving PDEs with PINNs

In our final example, we apply this framework to solving PDEs via Physics-Informed Neural Networks (PINNs). PINNs have recently shown great potential and garnered great interest for approximating solutions of PDEs [42, 105, 112]. It is typical in PINNs to generate samples via Monte Carlo Sampling (MCS). Yet, this suffers from a number of limitations, including low accuracy.

We use this general framework combined with the adaptive basis viewpoint [35] to devise a new adaptive sampling procedure for PINNs. Here, a Deep Neural Network (DNN) with n nodes in its penultimate layer is viewed as an element of the linear subspace spanned the n functions defined by this layer's nodes. Our method then proceeds as follows. First, we use an initial set of m = m 1 samples and train the corresponding PINN Ψ = Ψ 1 . Then, we use the adaptive basis viewpoint to construct a subspace F 1 with Ψ 1 ∈ F 1 . Next, we draw m 2 -m 1 samples using CS for the subspace F 1 , and use the set of m = m 2 samples to train a new PINN Ψ = Ψ 2 , using the weights and biases of Ψ 1 as the initialization. We then repeat this process, alternating between generating new samples via CS and retraining the network, to obtain a sequence of PINNs Ψ 1 , Ψ 2 , . . . approximating the solution of the PDE. We term this procedure Christoffel Adaptive Sampling (CAS) . See Appendices D.1-D.4 for further information.

100

0.8

1.6

0.8

2.5

0.6

0.6

1.4

0.4

0.4

1.2

test error

10·1

0.2

0.2

0

-0.2

0.8

-0.2

-0.4

0.6

-0.4

10-2

-tanh 5 × 50-MC

-sigmoid 5 × 50-MC

+tanh 5 × 50-CAS

sigmoid 5 × 50-CAS

-0.6

-0.8

0.4

-0.6

-0.8

0.5

0.2

2000

4000

6000

8000

10000

m

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

In Fig. 3 we show that this procedure gives a significant benefit over MCS in terms of the number of samples needed to reach a given accuracy. See Appendix D.5 for further details on the experimental setup. Note that both approaches use the same DNN architecture and are trained in exactly the same way (optimizer, learning rate schedule, number of epochs). Thus, the benefit is fully derived from the sampling strategy. The PDE considered (Burger's equations) exhibits shock formation as time increases. As can be seen in Fig. 3, CAS adapts to the unknown PDE solution by clustering samples near this shock, to better recover the solution than MCS.

As we explain in Appendix D.2, this example also justifies C > 1 in our general framework, as we have three sampling operators related to the PDE and its initial and boundary conditions. We note, however, that this example falls outside our theoretical analysis, since the sampling operator stemming from the PDE is nonlinear and the method is implemented in an adaptive fashion. In spite of this, however, we still see a nontrivial performance boost from the CS-based scheme.

4 Theoretical analysis

In this section, we present our theoretical analysis. Proofs can be found in Appendix E.

4.1 Sample complexity

We first establish a sufficient condition for empirical nondegeneracy (2.5) to hold in terms of the numbers of samples m c and the generalized Christoffel functions of suitable spaces.

Definition 4.1 (Subspace covering number) . Let ( X , ∥·∥ X ) be a quasisemi-normed vector space, F be a subset of X , n ∈ N 0 and t ≥ 0 . A collection of subspaces F 1 , . . . , F d of X of dimension at most n is a ( ∥·∥ X , n, t ) -subspace covering of F if for every f ∈ F there exists a j ∈ { 1 , . . . , d } and a f j ∈ F j ∩ F such that ∥ f -f j ∥ X ≤ t . The subspace covering number N ( F , ∥·∥ X , n, t ) of F is the smallest d such that there exists a ( ∥·∥ X , n, t ) -subspace covering of F consisting of d subspaces.

In this definition, we consider a zero-dimensional subspace as a singleton { x } ⊆ X . Therefore, the subspace covering number N ( F , ∥·∥ X , 0 , t ) is precisely the classical covering number N ( F , ∥·∥ X , t ) . We also remark that if F = ∪ d j =1 F j is itself a union of d subspaces of dimension at most n , then N ( F , ∥·∥ X , n ′ , t ) = d for any t ≥ 0 and n ′ ≥ n .

We now also require the following notation. If F ⊆ X , we set B ( F ) = { f/ ∥ f ∥ X : f ∈ F \{ 0 }} .

Theorem 4.2 (Sample complexity for empirical nondegeneracy) . Consider the setup of Section 2.1. Let F ⊆ X 0 , n ∈ N 0 , 0 < ϵ, δ < 1 and F 1 , . . . , F d be a ( | | |·| | | , n, t ) -subspace covering of B ( F -F ) , where t = √ αϵ/ 2 and | | |·| | | 2 = ∑ C c =1 ess sup θ c ∼ ρ c {∥ L c ( θ c )( · ) ∥ 2 Y c /ν ( θ c ) } . Suppose that

where c ϵ/ 2 is as in (E.6) and ˜ F = ∪ d i =1 F i . Then (2.5) holds with probability at least 1 -δ .

This theorem gives the desired condition (4.1) for empirical nondegeneracy (2.5). It is interesting to understand when this condition can be replaced by one involving the function K c evaluated over F -F rather than its cover ˜ F . We now examine two important cases where this is possible.

Corollary 4.3 (Sample complexity for unions of subspaces) . Suppose that F -F = ∪ d j =1 F j is a union of d subspaces of dimension at most n ∈ N . Then (4.1) is equivalent to

Corollary 4.4 (Sample complexity for classical coverings) . Consider Theorem 4.2 with n = 0 . Then (4.1) is implied by the condition

Remark 4.5 Theorem 4.2 is formulated generally in terms of subspace coverings. This is done so that the scenarios covered in Corollaries 4.3 and 4.4 are both straightforward consequences. Our main examples in Section 3 are based on the (unions of) subspaces case. While this assumption is relevant for these and many other examples, there are some key problems that do not satisfy it. For example, in low-rank matrix (or tensor) recovery [27, 38, 108, 117], the space F of rankr matrices (or tensors) is not a finite union of finite-dimensional subspaces. But tight bounds for its covering number are known [26, 106], meaning it fits within the setup of Corollary 4.4.

4.2 Christoffel sampling

Theorem 4.2 reduces the question of identifying optimal sampling measures to that of finding weight functions ν c that minimize the corresponding essential supremum in (4.1). The following lemma shows that this is minimized by choosing ν c proportional to the generalized Christoffel function.

̸

Lemma 4.6 (Optimal sampling measure) . Let L , F , K ( F ) and κ ( F ) < ∞ be as in Definition 2.6. Suppose that for almost every θ ∈ D there exists a f ∈ F such that L ( θ )( f ) = 0 . Then

for any measurable ν : D → R that is positive almost anywhere and satisfies ∫ D ν ( θ ) d ρ ( θ ) = 1 . Moreover, this optimal value is attained by the function ν ⋆ ( θ ) = K ( F )( θ ) /κ ( F ) , θ ∈ D .

In view of this lemma, to optimize the sample complexity bound (4.1) we choose sampling measures

As noted, we term this Christoffel Sampling (CS) . This leads to a sample complexity bound

In the case of subspaces (or union thereof with small d ), this approach leads to a near-optimal sample complexity bound for the total number of measurements m := m 1 + · · · + m C .

Corollary 4.7 (Near-optimal sampling for unions of subspaces) . Suppose that F -F = ∪ d j =1 F j is a union of d subspaces of dimension at most n ∈ N . Then ∑ C c =1 κ c ( F -F ) ≤ βdn . Therefore, choosing the sampling measures as in (4.2) with ˜ F = F -F and the number of samples m c ≤ c ϵ/ 2 · α -1 · κ c ( F -F ) · log(2 nd/δ ) leads to the overall sample complexity bound

We note in passing that ∑ C c =1 κ c ( F -F ) ≥ αn/p whenever dim( Y c ) ≤ p ∈ N , ∀ c . See Theorem E.4. Therefore, the sample complexity bound is at least O ϵ,α,p ( n log( nd/δ )) .

It is worth comparing these bounds with MCS. Suppose that the ρ c 's are probability measures, in which case MCS is well defined. For MCS, we have ν c ≡ 1 , ∀ c , and therefore the corresponding measurement condition is

where λ c ( ˜ F ) = ess sup θ ∼ ρ c K c ( ˜ F )( θ ) ≥ κ c ( ˜ F ) . Comparing with (4.3), we conclude that the benefit of CS over MCS in terms of sample complexity corresponds to the difference between the supremum of the Christoffel function (i.e., λ c ( ˜ F )) and its average (i.e., κ c ( ˜ F ) ). Thus, if K c has sharp peaks - as it does, for instance, in Fig. 2 - one expects a significant improvement from CS over MCS.

4.3 Generalization error bound and noisy data

Thus far, we have derived CS and shown parts (i) and (iii) of Theorem 2.7. In this section we establish the generalization bound in part (ii). For additional generality, we now consider noisy samples

where the e ic represent measurement noise (see Remark E.6 for some further discussion on the noise term). We also consider inexact minimizers. Specifically, we say that ˆ f ∈ F is a γ -minimizer of (2.2) for some γ > 0 if it yields a value of the objective function that is within γ of the minimum value. For example, ˆ f may be the output of some training algorithm used to solve (2.2).

Theorem 4.8. Let 0 < ϵ, δ, γ < 1 and consider the setup of Section 2.1, except with noisy data (4.4) . Suppose that (4.1) holds and also that ρ c ( D c ) < ∞ , ∀ c . Then, for any f ∗ ∈ X 0 and σ ≥ ∥ f ∗ ∥ X , the estimator ˇ f = T σ ( ˆ f ) , where ˆ f is a γ -minimizer of (2.2) , satisfies

5 Conclusions, limitations and future work

We conclude by noting several limitations and areas for future work. First, in this work, we have striven for breadth - i.e., highlighting the efficacy of CS4ML across a diverse range of examples - rather than depth. Our aim is to show that the well-known ideas for active learning in standard regression (e.g., leverage score sampling) can be extended to a very general setting. We do not claim that CS is the best possible method for each example, and as such, our experimental results only compare against (inactive) MCS. In each application considered, there are other domain-specific strategies that are known to outperform MCS. See [5, 6, 22, 55, 74, 104, 111] and references therein for Fourier imaging, and [10, 31, 52, 51, 86, 113] in the case of PINNs. CS is a general framework, and is arguably more theoretically grounded and less heuristic than some such methods. Nonetheless, further investigation is needed to ascertain which method is best in each setting. In a similar vein, while Figs. 1-3 show significant performance gains from our method, the extent of such gains depends heavily on the problem. In Appendices B.6 and D.6 we discuss cases where the gains are far more marginal. In the PINNs example in particular, additional studies are needed to see if CAS leads to benefits across a wider spectrum of PDEs. Moreover, there is the intriguing possibility of using CS as the starting point for more advanced active learning schemes - for example, by generalizing the linear-sample sparsification techniques of [32] or the 'boosting' techniques of [58].

Second, as noted previously, a limitation of our theoretical analysis is the log-linear scaling with respect to the number of subspaces d (see Theorem 2.7, part (iii)). While this can be overcome in cases such as sparse regression (see Appendix A.4), we expect a more refined theoretical analysis may be able to tackle this problem in the general setting. Another limitation of our analysis is that the sample complexity bound in Theorem 4.2 involves K c evaluated over the subspace cover ˜ F , rather than simply F -F itself. We anticipate this can also be improved through a more sophisticated argument. This would help close the theoretical gap in the generative models example considered in this paper. Another interesting theoretical direction involves reducing the sample complexity from log-linear to linear (when F is a linear subspace), by extending, for example, [32].

Finally, we reiterate that our framework and theoretical analysis are both very general. Consequently, there are many other potential problems to which we can apply this work. As noted, the main practical hurdle in applying this framework to other problems is to determine how to sample from the optimal measure (2.7), or some computationally tractable surrogate. Some other problems of interest include low-rank matrix or tensor recovery, as mentioned briefly in Remark 4.5, sparse regression using random feature models, as was recently developed in [64], active learning for single neuron models, as developed in [50], and operator learning [17, 73]. These are interesting avenues for future work.

Acknowledgments and Disclosure of Funding

BA acknowledges the support of the Natural Sciences and Engineering Research Council of Canada of Canada (NSERC) through grant RGPIN-2021-611675. ND acknowledges the support of Florida State University through the CRC 2022-2023 FYAP grant program.

References

B. Adcock, S. Brugiapaglia, and C. G. Webster. Sparse Polynomial Approximation of HighDimensional Functions . Comput. Sci. Eng. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2022.

B. Adcock and J. M. Cardenas. Near-optimal sampling strategies for multivariate function approximation on general domains. SIAM J. Math. Data Sci. , 2(3):607-630, 2020.

B. Adcock, J. M. Cardenas, and N. Dexter. CAS4DL: Christoffel adaptive sampling for function approximation via deep learning. Sampl. Theory Signal Process. Data Anal. , 20(21):1-29, 2022.

B. Adcock, J. M. Cardenas, N. Dexter, and S. Moraga. Towards optimal sampling for learning sparse approximation in high dimensions. In A. Nikeghbali, P. Pardalos, A. Raigorodskii, and T. M. Rassias, editors, High Dimensional Optimization and Probability , volume 191. Springer Optim. Appl., 2022.

B. Adcock and A. C. Hansen. Compressive Imaging: Structure, Sampling, Learning . Cambridge University Press, Cambridge, UK, 2021.

B. Adcock, A. C. Hansen, C. Poon, and B. Roman. Breaking the coherence barrier: a new theory for compressed sensing. Forum Math. Sigma , 5:e4, 2017.

H. K. Aggarwal, M. P. Mani, and M. Jacob. MoDL: Model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging , 38(2):394-405, 2018.

A. Alaoui and M. W. Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

A. K. Alekseev, I. M. Navon, and M. E. Zelentsov. The estimation of functional uncertainty using polynomial chaos and adjoint equations. Internat. J. Numer. Methods Fluids , 67(3):328341, 2011.

A. C. Aristotelous, E. C. Mitchell, and V. Maroulas. ADLGM: An efficient adaptive sampling deep learning Galerkin method. J. Comput. Phys , page 111944, 2023.

H. Avron, M. Kapralov, C. Musco, C. Musco, A. Velingker, and A. Zandieh. Random Fourier features for kernel ridge regression: approximation bounds and statistical guarantees. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 253-262. PMLR, 2017.

H. Avron, M. Kapralov, C. Musco, C. Musco, A. Velingker, and A. Zandieh. A Universal Sampling Method for Reconstructing Signals with Simple Fourier Transforms. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , STOC 2019, pages 1051-1063, New York, NY, USA, 2019. Association for Computing Machinery.

H. H. Barrett and K. J. Myers. Foundations of Image Science . Wiley-Interscience, Hoboken, NJ, 2004.

C. Basdevant, M. Deville, P. Haldenwang, J. Lacrouix, J. Ouazzani, R. Peyret, P. Orlandi, and A. Patera. Spectral and finite difference solutions of the Burgers equation. Comput. Fluids , 14(1):23-41, 1986.

A. Berk. Deep generative demixing: Error bounds for demixing subgaussian mixtures of Lipschitz signals. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4010-4014. IEEE, 2021.

A. Berk, S. Brugiapaglia, B. Joshi, Y. Plan, M. Scotte, and O. Yilmaz. A coherence parameter characterizing generative compressed sensing with Fourier measurements. IEEE J. Sel. Areas Inf. Theory , 3(3):502-512, 2023.

K. Bhattacharya, N. Hosseini, B. Kovachki, and A. Stuart. Model reduction and neural networks for parametric PDEs. J. Comput. Math. , 7:121-157, 2021.

J. Bigot, C. Boyer, and P. Weiss. An analysis of block sampling strategies in compressed sensing. IEEE Trans. Inform. Theory , 62(4):2125-2139, 2016.

J. Blechschmidt and O. G. Ernst. Three ways to solve partial differential equations with neural networks-a review. GAMM-Mitteilungen , 44(2):e202100006, 2021.

A. Bora, A. Jalal, E. Price, and A. G. Dimakis. Compressed sensing using generative models. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 537-546. PMLR, 06-11 Aug 2017.

C. A. Bouman. Foundations of Computational Imaging: A Model-Based Approach . SIAM, Philadelphia, PA, 2022.

C. Boyer, P. Weiss, and J. Bigot. An algorithm for variable density sampling with blockconstrained acquisition. SIAM J. Imaging Sci. , 7(2):1080-1107, 2014.

S. Brugiapaglia, S. Dirksen, H. C. Jung, and H. Rauhut. Sparse recovery in bounded Riesz systems with applications to numerical methods for PDEs. Appl. Comput. Harmon. Anal. , 53:231-269, 2021.

S. L. Brunton and J. N. Kutz. Data-Driven Science and Engineering . Cambridge University Press, 2 edition, 2022.

S. L. Brunton and J. N. Kutz. Machine learning for partial differential equations. CoRR , abs/2303.17078, 2023.

E. J. Candès and Y. Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Trans. Inform. Theory , 57(4):2342-2359, 2011.

E. J. Candès and B. Recht. Exact matrix completion via convex optimization. Found. Comput. Math. , 9(6):717-772, 2009.

C. Canuto, M. Y. Hussaini, A. Quarteroni, and T. A. Zang. Spectral methods: Fundamentals in Single Domains . Springer, 2006.

S. Chatterjee and A. S. Hadi. Influential observations, high leverage points, and outliers in linear regression. Statist. Sci. , 1(3):379-415, 1986.

S. Chen, R. Varma, A. Singh, and J. Kovaˇ ccevi´ c. A statistical perspective of sampling scores for linear regression. In 2016 IEEE International Symposium on Information Theory (ISIT) , pages 1556-1560, 2016.

X. Chen, J. Cen, and Q. Zou. Adaptive trajectories sampling for solving PDEs with deep learning methods. arXiv:2303.15704 , 2023.

X. Chen and E. Price. Active regression via linear-sample sparsification. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 663-695. PMLR, 2019.

I.-Y. Chun and B. Adcock. Compressed sensing and parallel acquisition. IEEE Trans. Inform. Theory , 63(8):4860-4882, 2017.

A. Cohen and G. Migliorati. Optimal weighted least-squares methods. SMAI J. Comput. Math. , 3:181-203, 2017.

E. C. Cyr, M. A. Gulian, R. G. Patel, M. Perego, and N. A. Trask. Robust training and initialization of deep neural networks: An adaptive basis viewpoint. In Jianfeng Lu and Rachel Ward, editors, Proceedings of The First Mathematical and Scientific Machine Learning Conference , volume 107 of Proceedings of Machine Learning Research , pages 512-536, Princeton University, Princeton, NJ, USA, 2020. PMLR.

W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszcz, and R. Pascanu. Sobolev training for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

D. D˜ ung, V . Temlyakov, and T. Ullrich. Hyperbolic Cross Approximation . Adv. Courses Math. CRM Barcelona. Birkhäuser, Basel, Switzerland, 2018.

M. A. Davenport and J. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE J. Sel. Topics Signal Process. , 10(4):602-622, 2016.

T. De Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

P. Deora, B. Vasudeva, S. Bhattacharya, and P. M. Pradhan. Structure preserving compressive sensing MRI reconstruction using generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 522523, 2020.

M. Derezinski, M. K. K Warmuth, and D. J. Hsu. Leveraged volume sampling for linear regression. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

W. E, J. Han, and A. Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat. , 5(4):349-380, 2017.

M. Eigel, R. Schneider, and P. Trunschke. Convergence bounds for empirical nonlinear least-squares. ESAIM Math. Model. Numer. Anal. , 56(1):79-104, 2022.

M. S. Eldred, L. W. T. Ng, M. F. Barrone, and S. P. Domino. Multifidelity uncertainty quantification using spectral stochastic discrepancy models. In Roger Ghanem, David Higdon, and Houman Owhadi, editors, Handbook of Uncertainty Quantification , pages 991-1036. Springer, Cham, Switzerland, 2017.

C. L. Epstein. Introduction to the Mathematics of Medical Imaging . Other Titles in Applied Mathematics. Society for Industrial and Applied Mathematics, 2nd edition, 2007.

T. Erdelyi, C. Musco, and C. Musco. Fourier sparse leverage scores and approximate kernel learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 109-122. Curran Associates, Inc., 2020.

M. Fanuel, J. Schreurs, and J. A. K. Suykens. Nyström landmark sampling and regularized Christoffel functions. Mach. Learn. , 111:2213-2254, 2022.

S. Foucart and H. Rauhut. A Mathematical Introduction to Compressive Sensing . Appl. Numer. Harmon. Anal. Birkhäuser, New York, NY, 2013.

A. Gajjar, C. Hegde, and C. P. Musco. Provable active learning of neural networks for parametric PDEs. In The Symbiosis of Deep Learning and Differential Equations II , 2022.

A. Gajjar, C. Musco, and C. Hegde. Active Learning for Single Neuron Models with Lipschitz Non-Linearities. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 4101-4113. PMLR, 2023.

Z. Gao, T. Tanf, L. Yan, and T. Zhou. Failure-informed adaptive sampling for PINNs, part II: combining with re-sampling and subset simulation. arXiv:2302.01529 , 2023.

Z. Gao, L. Yan, and T. Zhou. Failure-informed adaptive sampling for pinns. SIAM Journal on Scientific Computing , 45(4):A1971-A1994, 2023.

M. Geist, P. Petersen, M. Raslan, R. Schneider, and G. Kutyniok. Numerical solution of the parametric diffusion equation by deep neural networks. J. Sci. Comput. , 88:22, 2021.

X. Geng and L. Zeng. Gradient-enhanced deep neural network approximations. J. Mach. Learn. Model. Comput. , 3(4):73-91, 2022.

B. Gözcü, R. J. Mahabadi, Y.-H. Li, E. Ilcak, T. Çukur, J. Scarlett, and V. Cevher. Learningbased compressive MRI. IEEE Trans. Med. Imag. , 37(6):1394-1406, 2018.

L. Guo, A. Narayan, and T. Zhou. A gradient enhanced ℓ 1 -minimization for sparse approximation of polynomial chaos expansions. J. Comput. Phys. , 367:49-64, 2018.

L. Guo, A. Narayan, and T. Zhou. Constructing least-squares polynomial approximations. SIAM Rev. , 62(2):483-508, 2020.

C. Haberstich, A. Nouy, and G. Perrin. Boosted optimal weighted least-squares. Math. Comp. , 91:1281-1315, 2022.

K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. , 79(6):3055-3071, 2018.

J. Hampton and A. Doostan. Coherence motivated sampling and convergence analysis of least squares polynomial chaos regression. Comput. Methods Appl. Mech. Engrg. , 290:73-97, 2015.

J. Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. U.S.A. , 115(34):8505-8510, 2018.

B. Hanin. Which neural net architectures give rise to exploding and vanishing gradients? Advances in Neural Information Processing Systems , pages 582-591, 2018.

B. Hanin and D. Rolnick. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems , pages 571-581. 2018.

A. Hashemi, H. Schaeffer, R. Shi, U. Topcu, G. Tran, and R. Ward. Generalization bounds for sparse random feature expansions. Applied and Computational Harmonic Analysis , 62:310330, 2023.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

C. Heiß, I. Gühring, and M. Eigel. A neural multilevel method for high-dimensional parametric PDEs. In Advances in Neural Information Processing Systems , 2021.

A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir. Robust Compressed Sensing MRI with Deep Generative Priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 14938-14954. Curran Associates, Inc., 2021.

A. Jalal, S. Karmalkar, A. Dimakis, and E. Price. Instance-optimal compressed sensing via posterior sampling. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4709-4720. PMLR, 18-24 Jul 2021.

X. Jin, S. Cai, H. Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physicsinformed neural networks for the incompressible navier-stokes equations. J. Comput. Phys. , 426:109951, 2021.

J. Kaczmarzyk, P. McClure, W. Zulfikar, A. Rana, H. Rajaei, A. Richie-Halford, S. Bansal, D. Jarecka, J. Lee, and S. Ghosh. neuronets/nobrainer: 0.4.0, October 2022.

T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations , 2018.

D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. arXiv:1412.6980 , 2017.

N. Kovachki, Z. Li, B. Liu, K. Azizzadnesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. J. Mach. Learn. Res. , 24:1-97, 2023.

F. Krahmer and R. Ward. Stable and robust sampling strategies for compressive imaging. IEEE Trans. Image Process. , 23(2):612-622, 2013.

G. Kutyniok, P. Petersen, M. Raslan, and R. Schneider. A theoretical analysis of deep neural networks and parametric PDEs. Constr. Approx. , 55:73-125, 2022.

D. J. Larkman and R. G. Nunes. Parallel Magnetic Resonance Imaging. Phys. Med. Biol. , 52(7):R15, 2007.

J. B. Lasserre and E. Pauwels. The empirical Christoffel function with applications in data analysis. Adv. Comput. Math. , 45:1439-1468, 2019.

Y. Li, M. Anitescu, O. Roderick, and F. Hickernell. Orhogonal bases for polynomial regression with derivative information in uncertainty quantification. Int. J. Uncertain. Quantif. , 1(4):297320, 2011.

D. Liang, J. Cheng, Z. Ke, and L. Ying. Deep Magnetic Resonance image reconstruction: inverse problems meet neural networks. IEEE Signal Process. Mag. , 37(1):141-151, 2020.

Z. Liang and P. C. Lauterbur. Principles of Magnetic Resonance Imaging: A Signal Processing Perspective . IEEE Press Series on Biomedical Engineering. Wiley-IEEE Press, New York, 2000.

J. Liu and Z. Liu. Non-iterative recovery from nonlinear observations using generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 233-243, 2022.

B. Lockwood and D. Mavriplis. Gradient-based methods for uncertainty quantification in hypersonic flows. Comput. & Fluids , 85:27-38, 2013.

L. Lu, P. Jin, Z. Pang, G. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell. , 3:218-229, 2021.

A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos. Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Process. Mag. , 35(1):20-36, 2018.

P. Ma, M. W. Mahoney, and B. Yu. A statistical perspective on algorithmic leveraging. J. Mach. Learn. Res. , 16:861-911, 2015.

Z. Mao and X. Meng. Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions. Applied Mathematics and Mechanics , 44(7):1069-1084, 2023.

M. Mardani, E. Gong, J. Y. Cheng, S. S. Vasanawala, G. Zaharchuk, L. Xing, and J. M. Pauly. Deep generative adversarial neural networks for compressive sensing MRI. IEEE transactions on medical imaging , 38(1):167-179, 2018.

M. T. McCann and M. Unser. Biomedical image reconstruction: From the foundations to deep neural networks. Foundations and Trends® in Signal Processing , 13(3):283-359, 2019.

D. W. McRobbie, E. A. Moore, M. J. Graves, and M. R. Prince. MRI: From Picture to Proton . Cambridge University Press, Cambridge, 2nd edition, 2006.

G. Migliorati. Polynomial approximation by means of the random discrete L 2 projection and application to inverse problems for PDEs with stochastic data . PhD thesis, Politecnico di Milano, 2013.

G. Migliorati. Adaptive approximation by optimal weighted least squares methods. SIAM J. Numer. Anal , 57(5):2217-2245, 2019.

G. Migliorati. Multivariate approximation of functions on irregular domains by weighted least-squares methods. IMA J. Numer. Anal. , 41(2):1293-1317, 2021.

G. Migliorati, F. Nobile, and R. Tempone. Convergence estimates in probability and in expectation for discrete least squares with noisy evaluations at random points. J. Multivariate Anal. , 142:167-182, 2015.

C. Musco and C. Musco. Recursive Sampling for the Nyström Method. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

A. Naderi and Y. Plan. Beyond independent measurements: General compressed sensing with gnn application. In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems , 2021.

A. Narayan. Computation of induced orthogonal polynomial distributions. Electron. Trans. Numer. Anal. , 50:71-97, 2018.

P. Nevai. Géza Freud, orthogonal polynomials and Christoffel functions. A case study. J. Approx. Theory , 48(1):3-167, 1986.

T. O'Leary-Roseberry, P. Chen, U. Villa, and O. Ghattas. Derivative-informed neural operator: an efficient framework for high-dimensional parametric derivative learning. arXiv:2206.10745 , 2022.

T. O'Leary-Roseberry, U. Villa, P. Chen, and O. Ghattas. Derivative-informed projected neural networks for high-dimensional parametric maps governed by PDEs. Comput. Methods Appl. Mech. Engrg. , 388(1):114199, 2022.

G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory , 1(1):39-56, 2020.

B. Ordozgoiti, A. Matakos, and A. Gionis. Generalized leverage scores: Geometric interpretation and applications. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 17056-17070. PMLR, 2022.

B. Peherstorfer, K. Willcox, and M. Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. SIAM Rev. , 60(3):550-591, 2018.

J. Peng, J. Hampton, and A. Doostan. On polynomial chaos expansion via gradient-enhanced l 1 -minimization. J. Comput. Phys. , 310:440-458, 2016.

C. Poon. On the role of total variation in compressed sensing. SIAM J. Imaging Sci. , 8(1):682720, 2015.

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. , 378:686-707, 2019.

H. Rauhut, R. Schneider, and Z. Stojanac. Low rank tensor recovery via iterative hard thresholding. Linear Algebra Appl. , 524:220-262, 2017.

S. Ravishankar, J. C. Ye, and J. A. Fessler. Image reconstruction: from sparsity to data-adaptive methods and machine learning. Proc. IEEE , 108(1):86-109, 2020.

B. Recht, M. Fazel, and P. A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. , 52(3):471-501, 2010.

S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz. Data-driven discovery of partial differential equations. Sci. Adv. , 3(4), 2017.

C. M. Sandino, J. Y. Cheng, F. Chen, M. Mardani, J. M. Pauly, and S. S. Vasanawala. Compressed sensing: from research to clinical practice with deep neural networks. IEEE Signal Process. Mag. , 31(1):117-127, 2020.

F. Sherry, M. Benning, J. C. De los Reyes, M. J. Graves, G. Maierhofer, G. Williams, C.-B. Schönlieb, and M. J. Ehrhardt. Learning the sampling pattern for MRI. IEEE Trans. Med. Imag. , 39(12):4310-4321, 2020.

J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. J. Comput. Phys. , 375:1339-1364, 2018.

K. Tang, X. Wan, and C. Yang. Das: A deep adaptive sampling method for solving partial differential equations. arXiv preprint arXiv:2112.14038 , 2021.

T. Tang and T. Zhou. On discrete least-squares projection in unbounded domain with random evaluations and its application to parametric uncertainty quantification. SIAM J. Sci. Comput. , 36(5):A2272-A2295, 2014.

J. A. Tropp. User-friendly tail bounds for sums of random matrices. Found. Comput. Math. , 12:389-434, 2012.

M. Uecker. Parallel Magnetic Resonance Imaging. arXiv:1501.06209 , 2015.

M. Vidyasagar. An Introduction to Compressed Sensing . Comput. Sci. Eng. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2019.

S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. Sci. Adv. , 7(40):eabi8605, 2021.

D. P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science , 10(1-2):1-157, 2014.

J. Yu, L. Lu, X. Meng, and G. E. Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Comput. Methods Appl. Mech. Engrg. , 393:114823, 2022.

A Further literature and discussion

In this appendix, we provide some additional discussion that relates our work to existing literature.

A.1 Classical Christoffel functions

Let F be a linear subspace of the Hilbert space X = L 2 ρ ( D ) . The Christoffel function of F is the function θ ↦→ 1 /K ( F )( θ ) , where

̸

This is clearly a special case of Definition 2.6 with X 0 = C ( D ) and L : D → B ( X 0 , R ) given by L ( θ )( f ) = f ( θ ) . The Christoffel function is a classical object in approximation theory, most typically associated with the case where F is a space of algebraic polynomials [97]. Note that K ( F ) is precisely the diagonal of the reproducing kernel of F in X . It also follows immediately from Lemma E.1 that

for any orthonormal basis { f i } n i =1 of F .

A.2 Relation to leverage scores and leverage score sampling

Let D be a domain with a measure λ (typically the Lebesgue measure), F be a class of functions f : D → C and ρ be a probability measure over D with Radon-Nikodym derivative p = d ρ/ d λ . Then the leverage score is defined as

where ∥ f ∥ 2 p = ∫ D | f ( θ ) | 2 p ( θ ) d λ ( θ ) = ∫ D | f ( θ ) | 2 d ρ ( θ ) . See, e.g., [46] and references therein. Like with CS, leverage score sampling is a type of importance sampling where one draws samples randomly according to the probability density proportional to τ (or, often in practice, some easy-tocompute upper bound ˜ τ ). We remark in passing that leverage score sampling is sometimes known as effective resistance or coherence motivated sampling in the literature. Note that in the literature on leverage scores, the empirical nondegeneracy condition (2.5) is sometimes referred to as the subspace embedding condition (see, e.g., [119]).

It is readily seen that (A.1) a special case of Definition 2.6. Let X = L 2 ρ ( D ) . Then there are several different ways to formulate this.

(i) Consider D equipped with the measure λ and set L ( θ )( f ) = √ p ( θ ) f ( θ ) . Then nondegeneracy (2.3) holds with α = β = 1 and we have

̸

(ii) Alternatively, consider D equipped with the measure ρ and set L ( θ )( f ) = f ( θ ) . Then nondegeneracy (2.3) holds with α = β = 1 and we have

̸

Notice that K a ( θ ) = τ ( F , p )( θ ) is precisely the leverage score (A.1). Conversely, K b ( θ ) = τ ( F , p )( θ ) /p ( θ ) differs from the leverage score (A.1) by the factor p ( θ ) . Whether to include p ( θ ) in the definition is one of convention. What is important is that both (i) and (ii) lead to exactly the same Christoffel sampling sampling measure µ ⋆ in (2.11), which is equivalent to leverage score sampling for the standard regression problem. Indeed, this measure is

for case (i) or for case (ii).

(Statistical) leverage scores are an old concept [29] that have recently found many applications in machine learning, including randomized numerical linear algebra [119], kernel methods [8, 12, 46, 47, 94], active learning [12, 30, 41, 46, 85], data analysis [77] and beyond. Leverage scores are perhaps most commonly encountered in the discrete setting. Here the matrix leverage score of a matrix a matrix A ∈ R N × n is defined as

̸

where a i is the i th row of A . This is also a special case of Definition 2.6. Indeed, let X = R N and Y = R , both equipped with the Euclidean inner product, D = { 1 , . . . , N } be equipped with the uniform measure ρ , L ( i )( x ) = x i for a vector x = ( x i ) N i =1 ∈ R N and

Then (2.3) holds with α = β = 1 (notice that ρ is the uniform measure, not the uniform probability measure) and the generalized Christoffel function satisfies

̸

̸

There are a number of generalizations of leverage scores in the literature. See, e.g., [101]. However, to the best of our knowledge, none of these are similar to the setting considered in this paper (in particular, Definition 2.6).

A.3 Other related work

Our general framework and results are related to recent work [43]. Here the authors consider approximating functions in arbitrary (linear or nonlinear) approximation spaces (model classes) via an empirical least-squares regression based on a semi-norm that depends on a (random) variable y ∈ Y ⊆ R d . In a similar spirit to Theorem 4.2 and Theorem 4.8, [43] establishes sample complexity and generalization error bounds, and then use the former to optimize the sampling measure. Specifically, [43, Thm. 2.8 and Cor. 2.9] is similar to Theorem 4.2 with d = 0 (i.e., the case of classical coverings as opposed to subspace coverings) and [43, Thm. 2.12] is similar to Theorem 4.8, albeit without the additional technical steps used in the proof of Theorem 4.8 to obtain an expected error bound involving only the X -norm on the left- and right-hand sides.

This framework is related to our framework with C = 1 sampling operators and with X being a Banach space of functions. In fact, it is somewhat more general in this case: (i) no inner product structure is assumed and, (ii) each measurement (which are implicit in the semi-norm in [43]) could belong to a different space. Note that one could readily incorporate (ii) into our framework, by making the target space Y c depend on θ c . However, (i) leads to fundamentally weaker sample complexity guarantees. For example, in the case of a linear subspace F , the bounds in [43] scale log-cubically in n = dim( F ) , i.e., like c · n 3 · log( n ) (see [43, Sec. 3.1]), whereas the results in this work are log-linear, i.e., c · n · log( n ) (see Corollary 4.7), and therefore near-optimal. A similar suboptimal log-cubic scaling also arises in the case where F is a nonlinear space of s -sparse expansions in an orthonormal basis (see [43, Sec. 3.2]). See Appendix A.4 for more on this example.

As noted, the root cause of this suboptimal scaling is (i). In our framework, we impose that X is a Hilbert space and each Y c is a semi-inner product space. This allows us to use matrix concentration techniques (specifically, the matrix Chernoff bound [115]) in combination with the technique of subspace coverings (Definition 4.1) to obtain better sample complexity bounds. We remark in passing that the special case of d = 0 in our main sample complexity bound Theorem 4.2 (which, as noted above, is similar to [43]) does not require an inner product structure, i.e., it holds if X is a Banach space and each Y c is a semi-normed vector space. However, if applied to the case of linear subspaces, it would also lead to suboptimal sample complexity estimates much as in [43, Sec. 3.1].

A.4 The case of sparse regression

In this section, we briefly discuss the application of our general framework to sparse regression. For convenience, we shall assume that X = L 2 ρ ( D ) is the space of weighted square-integrable functions f : D → C , X = C ( D ) , C = 1 , Y = Y 1 = C with the Euclidean inner product and L = L 1 is the pointwise evaluation operator L ( θ )( f ) = f ( θ ) . However, the following discussion can be readily generalized to general setting considered in Section 2.1.

Let Φ = { ϕ i } N i =1 ⊂ X be a collection of elements in X , where N ∈ N . We assume that this is a Riesz system , i.e., there exist constants 0 < a ≤ b < ∞ such that

Note that a = b = 1 if and only if Φ is an orthonormal system. Now let 1 ≤ s ≤ N and define

Then F is the set of all s -sparse expansions in the system Φ . In this case, we refer to (2.2) as a sparse regression problem. Notice that empirical nondegeneracy Eq. (2.5) is in this case equivalent to the Restricted Isometry Property (RIP) [48] for the matrix A = 1 √ m ( ϕ j (Θ i )) m,N i,j =1 ∈ C m × N .

Several remarks are in order. First, notice that F s -F s = F 2 s . Second, F s is union-of-subspaces model. Indeed, F s is precisely a union of d = ( N s ) subspaces, each corresponding to a particular choice of support set S ⊆ { 1 , . . . , N } , | S | = s . When N and s are large, this is a large number, which makes the bound in Corollary 4.7 meaningless.

However, in this case, we can obtain a more appealing bound. Notice that the generalized Christoffel function

It is a simple argument to see that

Therefore,

The function ˜ K c serves two purposes. First, it can be used to upper bound κ ( F s ) = ∫ D K ( F s )( θ ) d ρ ( θ ) , i.e., the term appearing in the estimate in Corollary 4.7. Second, it can also be used as a convenient surrogate from which to sample, as computing ˜ K ( θ ) is typically easier than computing the true Christoffel function K ( F s )( θ ) .

Suppose now that ∥ ϕ i ∥ L ∞ ρ ( D ) = ess sup θ ∼ D | ϕ i ( θ ) | 2 ≤ K for all i = 1 , . . . , N . In this case, Φ is known as a bounded Riesz system [23]. Then

Using this and the well-known estimate d = ( N s ) ≤ (e N/s ) s , we deduce from Corollary 4.3 the sample complexity bound

This is a substantial improvement on the bound implied directly by Corollary 4.7, which depends linearly on d = ( N s ) . It also improves the bound of [43, Sec. 3.2] from log-cubic to log-quadratic (recall the discussion in Appendix A.3). However, the reader will note that it is not optimal, since m scales log-quadratically in s . One can improve this bound to log-linear by using a more sophisticated argument specific to the sparse regression case. See, e.g., [4, 23, 48] and references therein. The

extent to which this argument can be extended to the most general setting of our framework is a problem for future work.

Unfortunately, many Riesz systems of practical interest are not bounded. In this case, one has the sample complexity bound where

The bound (A.3) grows log-quadratically in s , as desired (it can also be improved to log-linear as mentioned above). However, the constant ˜ κ may depend on N . We refer to [4] for an extensive discussion on this issue. It is notable that ˜ κ can be estimated numerically. In cases of interest such as sparse multivariate polynomial regression, it is either bounded or mildly growing in N [4].

B Polynomial regression with gradient-augmented data

In this appendix, we describe the example considered in Section 3.1. Let f ∗ : R d → R . In the gradient-augmented learning problem, we consider training data of the form

where (for convenience) ∂ 0 f ∗ = f ∗ and ∂ k f ∗ = ∂f ∗ /∂θ k otherwise.

B.1 Formulation in terms of the general sampling framework

We now describe how to formulate the gradient-augmented learning problem as a special case of the general framework. Let C = 1 , D = D 1 = ( R d , ρ ) where d ≥ 1 and ρ be the tensor-product Gaussian measure on R d , i.e. d ρ = (2 π ) -d/ 2 e -1 2 ∥ θ ∥ 2 2 d θ . Then we let

be the Sobolev space of weighted square-integrable functions with respect to ρ with weak first-order derivatives that are also square-integrable with respect to ρ . This is a Hilbert space with inner product

Next, we let X 0 = C 1 ( D ) and Y be the Hilbert space

where ⟨· , ·⟩ 2 is the Euclidean inner product. Then, we define the sampling operator L : D →B ( X 0 , Y ) as

Notice that

Thus, Assumption 2.2 holds for this example with α = β = 1 .

B.2 The approximation space: multivariate Hermite polynomials

The standard Hermite polynomials H k ( x ) are eigenfunctions of the Sturm-Liouville problem ( wu ′ ) ′ +2 kwu = 0 , where w ( x ) = e -x 2 . They are orthogonal on R with respect to this function, specifically,

They satisfy the three term recurrence with H 0 ( x ) = 1 , H 1 ( x ) = 2 x and

and their derivatives satisfy the relation H ′ 0 = 0 and

See, e.g., [28, Sec. 2.6.2] for standard properties of Hermite polynomials.

It is more convenient for our purposes to consider the standard Gaussian (probability) measure R , i.e.,

Then the orthonormal Hermite polynomials (sometimes known as the 'probabilists' version') are given by

Manipulating (B.2), we see that the orthonormal Hermite polynomials satisfy ψ 0 ( θ ) = 1 , ψ 1 ( θ ) = y and

Also, we have ψ ′ n = 1 √ 2 n n ! 1 √ 2 H ′ n ( θ/ √ 2) = 1 √ 2 n n ! √ 2 2 nH n -1 ( θ/ √ 2) , and therefore ψ ′ 0 = 0 and

Using this, we readily deduce that the functions

form an orthonormal basis of the space

i.e., the Sobolev space of square-integrable functions f : R → R (with respect to ρ ) with squareintegrable (weak) first derivatives.

We extend to d dimensions via tensorization. We let

and consider the tensor Hermite polynomials

Leveraging what we did in the one-dimensional case, we can then construct an orthonormal basis of the Sobolev space

as { Φ ν } ν ∈ N d 0 , where

We conclude by defining the approximation space F . Fix an index p ∈ N 0 and let

This is the so-called hyperbolic cross index set, which is particularly well suited for multivariate polynomial approximation [1, 37]. Using this, we define the linear approximation space

̸

Notice that Assumption 2.5 holds for this choice of F . Indeed, F contains the constant function f ( θ ) = 1 , ∀ θ ∈ R d , which satisfies L ( θ )( f ) = (1 , 0 , . . . , 0) ⊤ = 0 .

B.3 Numerical solution of the regression problem

We now describe the numerical solution of the regression problem (2.2) in this case. Let θ 1 , . . . , θ m ∈ R d be sample points and ν : D → R be measurable and finite and positive almost everywhere. Then this problem takes the form

where y ik = ∂ k f ∗ ( θ i ) + e ik is the k th component of the i th measurement of f ∗ . For convenience, we assume the ordering S = { ν 1 , . . . , ν n } . Let f ∈ F be arbitrary and write f = ∑ n i =1 c i Φ ν i and c = ( c i ) n i =1 . Then where

Also, let and set

Then

Thus, the algebraic least-squares problem

is equivalent to (B.7), in the sense that every solution of this problem yields a solution ˆ f = ∑ n i =1 ˆ c i Φ ν i of (B.7) and vice versa.

Notice (B.8) is an m ( d +1) × n algebraic least-squares problem, which can be easily solved using standard linear algebra software once the matrix A has been computed. This is done using the relations (B.4) and (B.5).

B.4 Christoffel sampling

We now explain how to perform CS in this example. Directly sampling from the measure (2.7) is potentially difficult for this problem, since a simple characterization of the generalized Christoffel function K ( F )( θ ) is lacking. However, we observe that

̸

where

since the functions Φ ν are orthonormal. Therefore, we can use ˜ K ( F )( θ ) to construct a sampling measure as

Since ess sup θ ∼ ρ { K ( F )( θ ) / ˜ ν ( θ ) } ≤ n , Corollary 4.3 implies that sampling from ˜ µ yields optimal, log-linear sample complexity. As it is so closely related to the generalized Christoffel function (see (B.9)), we continue to refer to this as Christoffel sampling.

One can sample from ˜ µ by observing that it is an additive mixture [34] of tensor-product probability measures, each of which can be sampled from efficiently [96]. However, we opt for a simpler approach based on [2, 92] involving finite grids and discrete measures. Let Z = { z i } N i =1 ⊂ D grid of N ∈ N points, with each point z i drawn i.i.d. from the measure ρ . Then, we now replace the measure ρ by the finite measure

After doing this, the Christoffel function becomes a discrete function supported on the grid Z , which can be expressed as

where { χ j } n j =1 is an orthonormal basis of F with respect to the discrete H 1 ρ ( R ) -inner product. Now define the matrix

Consider the QR factorization B = QR , where Q = ( q ij ) ∈ C ( d +1) N × n has orthonormal columns and R ∈ C n × n is upper triangular. Then the j th column of Q contains precisely the values ∂ k χ j ( z i ) , i = 1 , . . . , N , k = 0 , . . . , d . Using this, we get

Hence, we can compute the function ˜ K ( F ) over the grid Z and then use this to sample from the optimal measure µ , which in this case is the discrete measure with θ ∼ µ if

B.5 Additional experimental details

As described in Appendix B.4 we first draw a random grid from the measure ρ . In our experiments, we choose the number of grid points as N = 50 , 000 . Note that this grid is drawn once, prior to any subsequent computations. In Fig. 1 we compute a test error as the relative Sobolev norm error over this grid. This is done using the following expression:

Notice that and therefore

r cond(A)

1015

MCS

number cond(A)

1015

MCS

number cond(A)

108

MCS

CS

CS

CS

number

106

1010

1010

104

Condition

105

105

Condition

102

200

400

600

800

500

1000

1500

2000

2500

500

1000

1500

m

m

m

The experiments shown in Fig. 1 proceed as follows. First, we generate a sequence of hyperbolic cross index sets (B.6) of orders 1 ≤ p 1 < p 2 < · · · < p L . Second, for each l = 1 , . . . , L , we choose m = m l according to the scaling

where n = n l = | S p l | . Then, we draw m samples randomly using either MCS or CS (as described above), compute the approximation ˆ f using (B.8) and then compute the error using (B.12). We then repeat this process for a total of T = 25 trials.

The graphs in Fig. 1 and this appendix show the geometric mean (the main curve) and plus/minus one (geometric) standard deviation (the shaded region). The reason for using the geometric mean is because the errors are plotted in log-scale on the y -axis. See [1, Sec. A.1] for further rationale behind this choice of visualization.

These experiments were conducted on a 2012 MacBook Pro with a 2.7 GHz Intel Core i7 CPU and 16 GB of DDR3-1600 RAM running MATLAB R2020b. Total runtime for these experiments was roughly three days.

B.6 Further experiments and discussion

Let 0 ≤ α ′ ≤ β ′ < ∞ be the optimal constants in the empirical nondegeneracy condition (2.5). In this example, for a collection of sample points θ 1 , . . . , θ m , these are given by

(here we recall that F -F = F , since F is a subspace). By expanding f ∈ F in the orthonormal basis { Φ ν : ν ∈ S } we readily deduce that

Thus, these constants can be computed numerically. Moreover, their ratio β ′ /α ′ = cond( A ) is precisely the 2 -norm condition number of A . In Fig. 4, we plot cond( A ) for both schemes using the same experimental setup as in Fig. 1. As predicted by our theory, CS leads to a bounded condition number for all m , that is at most roughly 10 2 in magnitude. However, MCS leads to an exploding condition number. This indicates that the scaling used Eq. (B.13) is not sufficient to ensure a good generalization bound, which is the reason for the poor performance of MCS in Fig. 1.

To examine this further, in Fig. 5 we perform the following experiment. For each value of n = n l , l = 1 , . . . , L , we compute the minimum value of m = m l such that cond( A ) ≤ tol , where tol = 10 . This is done as follows. We first set m = n/ ( d +1) and compute cond( A ) . If cond( A ) > tol , we increment m by one, draw new samples, re-compute cond( A ) and check whether cond( A ) ≤ tol . If not, we repeat this step. Finally, we repeat the whole process for a total of T = 25 trials.

2000

MCS

2000

*MCS

2000

量

MCS

CS

CS

一

CS

cn log(n)

cnlog(n)

cnlog(n)

1500

nlog(n)/(d + 1)

1500

nlog(n)/(d+1)

1500

-nlog(n)/(d+1)

1000

m

1000

1000

500

500

500

100

200

300

400

500

200

400

600

800

1000

200

400

600

800

1000

n

n

n

In Fig. 5, we show the results of this computation for MCS and CS. We plot both the arithmetic mean (the main curve) and plus/minus one standard deviation (the shaded region). As is evident, the CS scheme leads to a much less severe scaling. Further, as predicted by our main result, this scaling behaves as m ∝ n log( n ) . On the other hand, MCS exhibits a severe scaling. When d = 2 , for examples, one needs over 2000 sample points whenever n ≥ 30 . Notice that this exceedingly poor scaling for polynomial regression with MCS has been widely documented in the non-gradient augmented setting [57, 90, 114]. These experiments show gradient-augmented sampling also suffers from a similar effect.

As noted in Section 5, a limitation of this approach is that the benefit it delivers depends very much on the problem setting. For example, suppose we change the domain D to the bounded hypercube D = [ -1 , 1] d and consider polynomial regression using orthonormal polynomials with respect to the uniform probability measure on D . These polynomials can be constructed using tensor-products of the one-dimensional Legendre polynomials, much as was done in the previous case with the Hermite polynomials. In particular, analogous recurrence relations to (B.2) and (B.3) hold [28, Sec. 2.3], which allows for efficient construction of the least-squares matrix A once more. In Fig. 6 we display the approximation errors and condition numbers for MCS and CS for this problem. The comparison with Fig. 1 and Fig. 4 is striking. Especially in higher ( d ≥ 4 ) dimensions MCS performs nearly as well as CS, both in terms of the approximation error and condition number. This behaviour is also reflected in scaling computations, which are shown in the third row of Fig. 6. In low dimensions, MCSsuffers from a worse scaling that CS, but it is nowhere near as severe as in the case of unbounded domains. Moreover, as the dimension increases the scalings become much closer. Note that these observations are well known in the non-gradient augmented setting [57, 90].

B.7 Variation involving C > 1

We conclude this appendix by briefly describing several variations on the gradient-augmented regression problem that take advantage of the multimodal ( C > 1 ) formulation of the general sampling framework.

Partial gradient samples

In some of the aforementioned applications, one may only be able to afford fewer gradient samples than function evaluations [56, 103]. This can be handled as followed. Let C = 2 , m 1 be the number of function only samples and m 2 be the number of gradient-augmented samples. Then we define X = H 1 ρ ( R d ) , D 1 = D 2 = R d , ρ 1 = ρ 2 = ρ , Y 1 = ( C , ⟨· , ·⟩ 2 ) , Y 2 = ( C d +1 , ⟨· , ·⟩ 2 ) and

where c 0 = 1 / √ 2 and c k = 1 otherwise. Then we have

Thus, empirical nondegeneracy holds (2.5) as before.

() 1

10-5

MCS

+CS

10-5

CS

MCS

10°

MCS

SO+

(

SO+

MCS

10-10

10-6

10-01

10-15

10-15

10-9

101

102

103

102

103

102

103

102

103

m

m

m

m

Condition number cond(A)

1015

MCSH

Condition number cond(A)

104

MCS

Condition number cond(A)

102

SOW--

1number cond(A)

102

MCS

+CS

SO+

CS

SO+

1010

102

101

101

105

100

100

100

200

400

600

800

1000

200

400

600

800

500

1000

1500

20002500

500

1000

1500

m

m

m

m

1200

MCS

1200

MCS

1200

MCS

1200

MCS

1000

+Cs

1000 CS

1000

CS

1000CS

800

- n log(n)/(d + 1)

-cnlog(n)

800

- nlog(n)/(d + 1)

-cnlog(n)

800

-nlog(n)/(d + 1)

-cnlog(n)

800 -- nlog(n)/(d+ 1)

-cnlog(n)

m600

m600

m

600

m

600

400

400

400

400

200

200

200

200

50

100

150

200

250

300

100

200

300

400

500

200

400

600

800

1000

0

200

400

600

8001000

n

n

n

Hierarchical sampling

A limitation of the standard CS is that it is not well suited to hierarchical approximation. Often in practice, rather than a single approximation, one may wish to compute a sequence of approximations ˆ f 1 , ˆ f 2 , . . . it a nested sequence of subspaces F 1 ⊂ F 2 ⊂ · · · using a nested sequence of samples { θ i } m 1 i =1 ⊂ { θ i } m 2 i =1 ⊂ · · · . Standard CS is not well suited to this problem, since when the subspace changes from F k to F k +1 , so does the generalized Christoffel function. Therefore the existing points are drawn from the wrong distribution for optimal sampling in the new subspace F k +1 .

A means to overcome this problem was presented in [91] (see also [2]). It can be cast into our general framework using C = n = dim( F ) sampling operators. These works consider the unaugmented learning problem. However, it is readily extended to the gradient-augmented setting.

The basic idea is to define the i th sampling measure as

Notice that this is a probability measure, since the Φ ν i are orthonormal in H 1 ρ ( R d ) , Then, assuming that m = kn for some k ∈ N , we draw m i = k samples i.i.d. according to µ i for each i = 1 , . . . , n , giving m samples in total. We omit the full analysis of this case, but we remark in passing that this leads to near-optimal sample complexity. This is due, in essence, to the fact that

where ˜ K ( F ) is as in (B.10). The key point is that (B.14) is well suited to hierarchical approximation, since it draws k samples corresponding to each basis function Φ ν i . If the subspace F = span { Φ ν i : i = 1 , . . . , n } is augmented to F ′ = span { Φ ν i : i = 1 , . . . , n ′ } , where n ′ ≥ n , then we can recycle the existing samples, and draw additional samples for the new basis functions Φ ν i , i = n +1 , . . . , n ′ . See [2, 91] for further details.

C Fourier imaging with generative models

In this appendix, we describe the example considered in Section 3.2. We consider a discrete imaging scenario where the object to recover is a d -dimensional image of size n ×··· × n . In this paper, we consider 3D imaging, i.e., d = 3 .

Let N = n d and f ∗ ∈ C N be the complex-valued vectorized image. Let F ∈ C N × N be the matrix of the d -dimensional Discrete Fourier Transform (DFT) of length n with normalization F ∗ F = I . In (subsampled) Fourier imaging, one selects m ≤ N frequencies (typically, m ≪ N ), which correspond to m rows of F . Let Ω ⊆ { 1 , . . . , N } , | Ω | = m , be the set of frequencies sampled. Then the noisy measurements of the unknown image f ∗ take the form

where e ∈ C m is noise and P Ω ∈ C m × N is the row selector matrix, i.e., P Ω F selects the rows of F corresponding to the indices in Ω .

C.1 Formulation in terms of the general sampling framework

Depending on the imaging modality, it may be possible to select individual frequencies according to some sampling strategy. However, in applications such as MRI, this may not be possible. Typically, the scanner is only allowed to sample along piecewise smooth curves in frequency space [80, 89]. Fortunately, as we now describe, both models can be cast in our general sampling framework.

Let X = X 0 = C N with the Euclidean inner product and norm and C = 1 . Let ∆ 1 , . . . , ∆ M ⊆ { 1 , . . . , N } be a partition of { 1 , . . . , N } , i.e., ∪ i M =1 ∆ i = { 1 , . . . , N } . For convenience, we assume the ∆ i have the same cardinality, i.e., | ∆ i | = p , ∀ i . Then we let D = { 1 , . . . , M } , ρ be the uniform probability measure on D and Y = C p with the Euclidean inner product. Next, we define the sampling operator by

In other words, L ( i ) is the linear operator that outputs the Fourier transform of an image x at the frequencies in the i th partition element ∆ i . Observe that Assumption 2.4 holds for this example with α = β = 1 . Indeed, for any x ∈ C N , we have

With this in hand, our goal now is to design a suitable sampling measure µ on D = { 1 , . . . , M } , so that each random draw from µ corresponds to a sample of frequencies from the corresponding partition element. Let us first describe two examples.

Example C.1 (Sampling individual frequencies) Suppose the modality allows for sampling one frequency at a time. Here, we let M = N and define ∆ i = { i } , i = 1 , . . . , N .

We consider Example C.1 later in Fig. 8. Fig. 2 in the main paper is, however, based on the following example.

Example C.2 (Sampling horizontal lines in k -space) Phase encoding is a common sampling strategy in MRI. Here one fully samples in the first k -space direction and subsamples in the others. Thus, the scanner traverses a piecewise smooth trajectory in k -space. Mathematically, this can be formulated as follows. Let d = 3 for concreteness (other values of d can be easily handled), N = n 3 and M = n 2 . 3D frequencies are naturally enumerated over the set of multi-indices { 1 , . . . , n } 3 . Now consider a partition ∆ ′ i 1 i 2 , i 1 , i 2 = 1 , . . . , n , of this set defined by

Thus, ∆ ′ i 1 i 2 consists of k -space values along a horizontal line with y -z coordinate ( i 1 , i 2 ) . We now let ∆ i 1 i 2 ⊂ { 1 , . . . , N } be the image of ∆ ′ i 1 i 2 with respect to whatever ordering is used to convert triples ( i, j, k ) ∈ { 1 , . . . , n } 3 to single indices in { 1 , . . . , N } (in our experiments, we use lexicographical ordering). Then the sets ∆ i 1 i 2 define a partition of { 1 , . . . , N } , as required.

Note that the above formulation is by no means limited to these two examples. Many other popular sampling strategies can equally well be considered. This includes radial line sampling, spiral sampling and various other strategies used in practice. We remark in passing that the setup described here is closely related to the block sampling setup in compressed sensing. See [18, 22].

Remark C.3 It is worth noting two extensions of this setup. First, in practice one may wish to consider overlapping partitions ∆ 1 , . . . , ∆ M . Here, one requires a small modification of the sampling operators to ensure Assumption 2.4 holds. For an index j ∈ { 1 , . . . , N } let r j ∈ N be the number of partition elements in which j appears, i.e., r j = |{ i : j ∈ ∆ i }| . Then define

We now argue as in (C.2) to deduce that empirical nondegeneracy holds with α = β = 1 .

Second, it is sometime useful in practice to allow the partition elements to have different cardinalities, i.e., | ∆ i | = p i , i = 1 , . . . , M . This can be handled straightforwardly by zero padding. Let Y = C p , where p = max i =1 ,...,M { p i } . Then we define L ( i ) : C N → C p by appending p -p i zeros to the vector defined in (C.1) (or (C.3) if the partition is overlapping).

C.2 The approximation space: compressive imaging with generative models

The approach considered in this paper is based on the work [20] which introduced the generative compressed sensing framework. Their subsequent work on posterior sampling further delved into the problem [68]. In compressed sensing, the primary objective is to leverage the inherent structure in the solutions of underdetermined systems of noisy linear measurements. Most of the existing literature in this field focuses on imposing various forms of sparsity, such as standard sparsity, joint sparsity, hierarchical sparsity, or block sparsity [5, 48]. However, the generative compressed sensing framework takes a different approach. It assumes that the solutions to these problems lie near the range of a generative model G : R p → R N . By employing a well-trained generative model G that is relevant to the reconstruction task at hand, one can solve an optimization problem to reconstruct images or other signals. In this case, the approximation space F = range( G ) ⊂ R N , i.e., the range of the generative model G . This space can be seen as a low-dimensional manifold embedded in a higher-dimensional space. Even for simple DNN models this space is typically nonlinear, and certainly for the complex models considered for the experiments related to this section.

Importantly, the generative compressed sensing framework is not limited to image reconstruction but has broader applicability. The recent work [15] extends the analysis from [20] to the problem of demixing with subgaussian matrices. Other notable contributions include variational networks [59], generative adversarial networks trained with least-squares and pixel-wise ℓ 1 /ℓ 2 cost functions [87], generative adversarial network frameworks for structure preserving compressive sensing MRI [40], and model-based approaches combined with regularization and unrolled network architectures [7]. Furthermore, a recent study considers the semi-parametric single-index model with a generative prior under a Gaussian measurement assumption [81]. These studies, along with other diverse works, demonstrate the impressive versatility of the generative compressed sensing approach.

C.3 The generalized Christoffel function

Let F ⊆ X , where, as stated above, X = C N with the Euclidean inner product. Denote the Euclidean norm by ∥ · ∥ . Then the generalized Christoffel function is defined by

for i ∈ { 1 , . . . , M } . Since D = { 1 , . . . , M } is a discrete set, we consider discrete probability measures. Denote such a measure by ( µ i ) i M =1 , i.e., I ∼ µ if P ( I = i ) = µ i . Then sampling according to the Christoffel function gives the values

where

Algorithm 1 Computing ˜ K

Note that the corresponding weight function is

In order to apply Theorem 4.2 to this example, we need to identify a (subspace) cover ˜ F = F 1 ∪· · ·∪ F d of the set F -F , where, as above F , is the range of a generative model. It is therefore important to first ask when the set F -F admits such a cover for a reasonable value of d (since d appears logarithmically in the bound). This question has been addressed in [95, Lem. 3.1] in the case of ReLU generative models (see also [16, Lem. S2.2]). Roughly speaking, if G : R p → R N has L layers and the widths of its hidden layers are O ( p ) , then F -F can be covered exactly with log( d ) ≲ pL subspaces of dimension 2 p .

There are two factors that prohibit direct application of Theorem 4.2 to this example. First, the generative models we use are not simple ReLU DNNs. Second, even if they were, identifying the aforementioned union of subspaces that form the cover and computing the constant K ( ˜ F ) is likely computationally infeasible, since d grows exponentially in pL .

As noted in Section 5, however, we believe the estimate involving K ( ˜ F ) in Theorem 4.2 to be non-sharp, and conjecture that it should suffice to consider K ( F -F ) instead. Therefore, in what follows, we compute with (an approximation to) the former. Thus, in this example, the theory does not completely apply. Nevertheless, as noted, we see significant benefits from CS for this example.

C.4 Numerical approximation to the generalized Christoffel function

In view of (C.4), computing K ( F -F ) involves solving M (nonlinear, nonconvex) maximization problems over the input space R p of the generative model G . This is extremely computationally expensive, since both p and M are large in practice. In our examples, M = n 3 (Theorem C.1) or M = n 2 (Theorem C.2), where n = 128 , and p = 1024 (see next).

Therefore, in this work we consider a different approach, which is based on an iterative, random sampling procedure for computing an approximation ˜ K to K . This procedure is presented in Algorithm 1 and involves repeated random samples from the input distribution of the generative model to construct candidate elements f 1 -f 2 ∈ F -F . It has the advantages of being simple and not computationally intensive. Moreover, as our examples show, sampling from the resulting ˜ K achieves the objective of significantly outperforming MCS.

The convergence of Algorithm 1 in computing ˜ K can be seen in Fig. 7 where the relative ℓ 2 error of the iterates are computed with the final ˜ K after the algorithm returns.

C.5 Additional experimental details

The experiments in Figures 2 & 8 were conducted using a single generative neural network model trained on brain MRI data. The braingen-0.1.0 generative model was obtained from https: //github.com/neuronets/trained-models/releases and was trained using the nobrainer

0

200

400

600

800

1000

10 -3

10 -2

10 -1

10 0

0

200

400

600

800

1000

10 -3

10 -2

10 -1

10 0

framework [70] available under Apache License, Version 2.0. This model was trained using a progressively growing strategy [71] for generation of T1-weighted brain MR scans. The model takes as input a latent vector v ∈ R 1024 , here generated by sampling from the standard normal distribution, i.e. v i ∼ N (0 , 1) , i = 1 , . . . , 1024 . The output dimension of the selected braingen-0.1.0 model is N = n 3 where n = 128 .

First we generate ˜ K using 1000 iterations of Algorithm 1. To generate the results, we chose an image from the range of the generator using random seed '12345'. We then ran 20 random trials reconstructing this image with the same computed ˜ K , solving problem (2.2) using 5000 iterations of the Adam optimizer [72], implemented in TensorFlow. A constant learning rate multiplier of 100 was used to improve the convergence in the gradient update step. The experiments were conducted on a single computer with an AMD Ryzen 9 5950x, 64 GB of DDR4-3600 RAM, and a NVIDIA GeForce RTX 3090 Ti. The total runtime for these experiments was approximately 2 weeks. In Fig. 2 we consider the setup of Example C.2. In Fig. 8 we consider the setup of Example C.1. As in Section B.5, our error plots display the mean PSNR value and also a shaded region corresponding to one standard deviation. The PSNR is defined as

where MSE is the Mean Squared Error

Here f ∗ is the ground truth image, ˆ f is its reconstruction and MAX f ∗ is the maximum possible pixel value of the image, here equal to 255 .

C.6 Further experiments and discussion

In Fig. 2 consider the strategy of sampling horizontal lines in k -space, see Example C.2. In Fig. 8 we perform the same experiment with the strategy of sampling individual frequencies in k -space, see Example C.1. While CS with line sampling performs similarly to CS with individual frequency sampling, MCS line sampling in k -space can be seen to have a 15-20% drop in image reconstruction quality over MCS individual frequency sampling in k -space.

Although, as noted, the theory does not directly apply to this example, it is worth examining the relevant constants to gain some further insight into the performance of CS over MCS. For MCS, Theorem 4.2 suggests that the sample complexity depends on the term

30

40

50

60

70

80

90

5

10

15

20

25

30

35

40

45

0.25

0.5

0.75

1.0

1.25

1.5

1.75

2.0

10

15

20

25

30

35

40

45

i.e., the maximum of the approximate Christoffel function ˜ K . Conversely, for CS it suggests that the sample complexity depends on its average, i.e.,

In Table 1 we report these values for the two experiments. As is evident, CS leads to a greatly reduced constant in both cases over MCS, which is in agreement with the general improvement in PSNR for this method. We note, however, that these constants do not capture all the phenomena seen in the experiments. For example, MCS performs worse for Example C.2 than it does for Example C.1. Yet the size of the constants would suggest otherwise. This is an interesting gap between the theory and practice.

C.7 Variations involving C > 1

The setup described so far (sampling the Fourier transform) is a discrete model for so-called singlecoil MRI. However, modern MR scanners are typically employ parallel coils. Here, multiple receive coils acquire Fourier measurements in parallel. Let C ≥ 1 be the number of coils. Then one can model the measurements in parallel MRI as

Here b c ∈ C m c are the measurements in the c th coil, Ω c ⊆ { 1 , . . . , N } with | Ω c | = m c and S c is a diagonal matrix, known as the sensitivity profile of the coil. See, e.g., [76, 116].

This is a multimodal sampling model that can be cast into our framework. Indeed, we replace the single sampling operator (C.1) with C sampling operators defined by

Note that empirical nondegeneracy (2.5) is easily shown to be equivalent to the condition

Sensitivity profiles are often designed so that this property holds with moderate α , β . Note that in practice the sensitivity profiles are not known, and have to be computed either via a calibration pre-scan or learned during the process of recovering the image. See, e.g., [5, Sec. 3.5.2-3.5.3] and references therein.

D Solving PDEs using PINNs

In this appendix, we provide further details for the example considered in Section 3.3 involving solving the nonlinear Burger's equation using PINNs. As mentioned, we implement CS in an adaptive manner for this example, using a strategy we term Christoffel Adaptive Sampling (CAS) . This strategy not only applies to the numerical solution of PDEs, but to any regression problem with DNNs and general, multimodal data. We then apply this to the numerical solution of PDEs through the framework of PINNs.

D.1 PINNs for solving PDEs

Consider a spatial domain D ⊂ R d and time interval I = [0 , T ] , where d ≥ 1 and T ≥ 0 , respectively. Then we seek to learn an unknown function u ∗ : D × [0 , T ] : → C given as the solution of a PDE of the form

Here L is a differential operator in the space and time variables, u i is a known initial condition and u b is a known boundary condition. Throughout this section, we use f to denote the right-hand side of the PDE and u ∗ to denote the target function (instead of f ∗ ), as is customary in the numerical PDE literature.

The general idea behind PINNs [69, 105] is to consider training data of the form

where ( x d i , t d i ) ∈ D × (0 , T ] are sample points in the interior of the domain, ( x i i , t i i ) ∈ D ×{ 0 } are sample points at the initial condition and ( x b i , t b i ) ∈ ∂D × (0 , T ] are sample points on the space-time boundary of the domain. Typically, these sample points are generated via MCS. One then seeks a minimizer of the problem

where U is a class of DNNs u : R d × R → R and λ i , λ b > 0 are hyperparameters that control fitting the PDE versus fitting the boundary/initial conditions. Note that we write U and u ∈ U throughout this section instead of F and f ∈ F , to avoid confusion with the right-hand side of the PDE.

D.2 PINNs formulation in terms of the general sampling framework

We now show how to reformulate PINNs in terms of the general sampling framework. Since there are three types of data, we let C = 3 . Then we define D 1 = D × (0 , T ) , D 2 = D × { 0 } and D 3 = ∂D × (0 , T ] , and let ρ 1 , ρ 2 and ρ 3 be measures on these domains. We then consider the sampling operators

Next, we let ν c : D c → R and µ c be as in Assumption 2.2. Finally, for each c = 1 , 2 , 3 , we draw m c samples (Θ ic ) m c i =1 , where Θ ic = ( X ic , T ic ) , and consider the weighted regression problem

There are several factors that prohibit direct application of CS to (D.2). The first is that the sampling operator L 1 is potentially nonlinear, since it involves the generally nonlinear PDE operator L . This means that there is no theoretical basis for the use of CS. However, even if it were linear (e.g., for a linear PDE), it could be very difficult to compute the generalized Christoffel functions K c ( U ) in practice, since evaluating K c ( U ) at a single point θ involves solving a maximization problem over the space U (a class of DNNs). The second, and arguably more critical issue, is that PINNs are typically implemented in the highly overparametrized regime, where the number of DNN parameters greatly exceeds the amount of training data. CS is designed to provide sampling measures that ensure 'good' recovery over the whole of the space U . But this is an extremely conservative approach for DNN-based learning problems, since only certain DNNs are likely to be encountered in training.

We tackle these problems in reverse order. For the second, we present an adaptive sampling approach for deep learning. This is described in the next subsection. Note that this does not assume the underlying problem is a PDE. Our presentation follows the lines of the general sampling framework. Having done this, we then describe its application to PINNs, including suitable modifications of the sampling measures and loss function that take into account the nonlinearity of the PDE operator.

D.3 CAS for deep learning with general, multimodal samples

In this section, we consider DL based on general, multimodal samples as introduced in our general sampling framework. What follows is an extension of a method first introduced in [3], which in turn is based on ideas from [35]. Let X be a Hilbert space of functions R d 1 → R d 2 and U ⊂ X be a class of DNNs of a fixed architecture. As in Section 2, we consider sampling operators

where the Y c are Hilbert spaces. We also write u ∗ ∈ X for the unknown function we seek to learn.

We now describe the adaptive sampling method. This exploits the so-called adaptive basis viewpoint of DNNs, as developed in [35]. Let n ∈ N be the number of nodes in the penultimate layer of the network architecture used to define U and, for convenience, assume that the output is not subject to a bias term, just a weight. Therefore, we can write any u ∈ U as

where H : R d 1 → R n is a DNN of the same architecture as u , except for the final layer, and C = ( c ij ) n,d 2 i,j =1 is the weight matrix on the output layer. Now let u i : R d 1 → R be the i th component of u , i = 1 , . . . , d 2 , and H i : R d 1 → R be the i th component of H , i = 1 , . . . , n . Then we can express h j as

In particular, each u j is an element of the subspace span { H 1 , . . . , H n } .

Now consider an initial DNN ˆ u = ˆ u (0) = (ˆ u (0) j ) d 2 j =1 obtained via a standard (random) initialization strategy. Write ˆ u as in (D.3) as

let ̂ H (0) = ( ̂ H (0) i ) n i =1 and consider the subspace

so that ˆ u (0) j ∈ H (0) , j = 1 , . . . , d 2 . Let p 0 = dim( H (0) ) ≤ n . We now perform CS using this subspace. In other words, we define sampling measures

then, for each c , we generate m (1) c sample points Θ ic i.i.d. from µ (1) c and generate samples as

Given these samples, we then train a new DNN by applying a standard optimizer to the minimization problem

Let ˆ u (1) ∈ U be resulting DNN. The idea now is to repeat this process by drawing m (2) c -m (1) c new samples using CS for the new subspace H (1) and then using the full set of m (2) c samples to compute a new DNN ˆ u (2) . Continuing in this way, we generate a sequence of DNNs ˆ u (1) , ˆ u (2) , . . . ∈ U approximating the unknown function u ∗ . To be precise, consider step l ≥ 1 . Let ˆ u ( l -1) be given, write and define

We use this to construct sampling measures

and then, for each c , we generate m ( l ) c -m ( l -1) c new sample points Θ ic , i = m ( l -1) c +1 , . . . , m ( l ) c , i.i.d. from µ ( l -1) c and corresponding samples

We then combine these with the existing m ( l -1) c samples, giving a total of m ( l ) c samples for each c , and then use this to train a new DNN via the minimization problem

We write ˆ u ( l ) ∈ U for the resulting DNN.

D.4 CAS for PINNs

Our aim now is to apply the CAS method in the previous subsection to PINNs. In this case, we set d 1 = d +1 and d 2 = 1 . As we now describe, we make one main modification to CAS in this case, which is to replace the true generalized Christoffel functions K c with surrogates ˜ K c that are more amenable to computations.

Recall that C = 3 in this case and that the sampling operators are as in (D.1). To define ˜ K 1 we simply forget the differential operator L and define

Next, for the initial ( c = 2 ) and boundary ( c = 3 ) data, we define

and

Let ˜ ν ( l ) c and ˜ µ ( l ) c denote the corresponding weight function and sampling measures, i.e.,

Then at step l of the procedure, we generate m ( l ) c -m ( l -1) c new sample points ( X ic , T ic ) , i = m ( l -1) c +1 , . . . , m ( l ) c , i.i.d. from ˜ µ ( l -1) c and compute the corresponding samples

where the L c are as in (D.1). We then combine these with the existing m ( l -1) c samples and use this to train a new DNN via the minimization problem

We term this method CAS for PINNs . It is summarized in Algorithm 2.

Steps 2-3 of this algorithm require some further explanation. First, as in previous examples (see Appendix B), we introduce finite grids Z c = { z ic } N c i =1 , c = 1 , 2 , 3 , to represent the domains D c , c = 1 , 2 , 3 . This is to enable efficient sampling from the measures ˜ µ c , which are now discrete measures defined over the corresponding grids. See the next section for the construction of these grids.

Second, recall from Lemma E.1 that in order to compute the functions ˜ K c , we need to construct orthonormal bases for space

over the finite grids Z ic . Previously, in Appendix B.4, we did this via QR factorization. However, the dictionary { ̂ H ( l -1) i } N i =1 generated in Step 1 of the algorithm is often numerically redundant. Therefore, in this case we use a truncated SVD instead. Specifically, if

then we compute its SVD U c Σ c V ∗ c and define the numerical dimension

Using this, we now obtain an orthonormal basis over the grid Z c using the first p l -1 ,c left singular vectors, i.e., the first p l -1 ,c columns of U c . Given this basis, we can then compute the function ˜ K c ( H ( l -1) ) over the grid via Lemma E.1. To precise,

and the corresponding measures ˜ µ c are given by θ ∼ ˜ µ c if

D.5 Additional experimental details

In this section, we describe the setup for our experiments. The problem we consider is the Burger's equation with D = ( -1 , 1) , T = 1 ,

and initial and boundary conditions

Algorithm 2 CAS for PINNs

input: PDE operator L , initial and boundary condition u i and u b . Class of DNNs U with fixed architecture, no bias on the output layer and penultimate layer of with N . Finite grids Z c = { z ic } N c i =1 ⊂ D c , where N c ≥ N , c = 1 , 2 , 3 , numbers of samples 0 = m (0) c < m (1) c < m (2) c < . . . , c = 1 , 2 , 3 , and tolerance δ tol > 0 .

initialize: Initialize the DNN ̂ h (0)

for l = 1 to t do

.

Construct the dictionary { ̂ H ( l -1) 1 , . . . , ̂ H ( l -1) N } via the relation

Construct the sampling measures ˜ µ ( l ) c , c = 1 , 2 , 3 , as in (D.4).

For each c = 1 , 2 , 3 , draw m ( l ) c -m ( l -1) c sample points ( X i,c , T i,c ) , i = m ( l -1) c +1 , . . . , m ( l ) c . Train the DNN ˆ u ( l ) ∈ U by applying a standard optimizer to (D.5) with initialization ˆ u ( l -1) .

end for output: A sequence of DNN approximations ̂ u (1) , ̂ u (2) , . . . , ̂ u ( t ) to u .

and

respectively. As mentioned, we employ three finite grids

Following Appendix B, these are generated once beforehand by MCS with respect to the uniform measure over each domain.

We now describe further aspects of our experiments.

Methods considered. As in previous appendices, we compare CS with MCS. Specifically, we consider DNNs trained using training data constructed via CAS (i.e., Algorithm 2), and DNNs trained from standard training data obtained via MCS from the uniform measure over each domain. As in previous example, we labels these as 'CS' and 'MCS', respectively. For Algorithm 2 we use the threshold parameter δ tol = 10 -6 , since our computations are performed in single precision.

Choice of architectures and initialization. Weconsider fully-connected DNNs with activation function ρ , L ≥ 1 hidden layers and an equal number of N ≥ 1 nodes per hidden layer. We term these ρ L × N DNNs. Since we are solving a regression problem, we assume the output layer is not subject to ρ . We also set the bias to be zero in this layer. For the activation function ρ , we consider either the Rectified Linear Unit (ReLU), hyperbolic tangent (tanh), Exponential Linear Unit (ELU), or sigmoid. These are defined as follows:

ReLU:

tanh:

ELU:

sigmoid:

ρ

(

x

) :=

{

x

:

x

Note that we choose DNNs with a fixed number of nodes N for each layer. Moreover, we also focus on DNNs with depth L such that the ratio L/N is small. This is motivated by results in [62]. To be precise, L/N = 0 . 1 in our experiments.

For the initialization strategy, we consider weights and biases with entries drawn from the Normal initialization with mean 0 and variance 0 . 1 . Having in mind the size of architectures described in this

0

,

(

e

x

-

1) :

x <

0

}

,

work, this initialization leads to smaller variance than commonly used initializations used PINNs such as the well-known Xavier and He Normal initializations. Besides, it has been shown that is an effective choice for function approximation, see [63, 65]. In all our experiments, the DNNs were initialized with the same seed to compare the effect of varying the samples.

Optimizers for training and parametrization. We use Adam as the optimizer in combination with an exponentially decaying learning rate with respect to the number of epochs. For both CAS and the procedure based on MCS, we train the networks following a schedule number of epochs, described as follows:

For the CAS strategy, we train the networks for n ( l ) epochs on each new set of samples, using the weights and biases from the previous set as initialization. We repeat this procedure 9 times, i.e. we train for a total of 225 , 000 epochs. For MCS, we train the network for a total of 225 , 000 epochs as well, stopping every n ( l ) epochs to add more data points drawn from the uniform distribution, and continuing to train from the previous point.

Training data and design of experiments. To measure the performance of these methods, we present the average testing error and run statistics over 20 trials for both sampling strategies. We train our DNNs over a set of training data consisting of the values

For each c = 1 , 2 , 3 , we generate each data set of size m ( l ) c , with 0 < m (1) c < . . . < m ( l ) c < . . . < m (9) c , we sample 20 i.i.d. sets of points { Θ ic } m (9) c i =1 from the sampling measures for each strategy. The precise values for the numbers of samples m ( l ) c are given in Table 2.

Testing data and error metric. Let u ∗ be the solution of the PDE and ˆ u be a DNN approximation to it. We compute the (discrete) relative error

where Z t = Z t 1 ∪ Z t 2 ∪ Z t 3 and M = M 1 + M 2 + M 3 . Here, for each c = 1 , 2 , 3 , Z t c is a test grid of size M c , with points drawn randomly from the uniform measure over the corresponding domain. We set M 1 = 8000 , M 2 = 1000 and M 3 = 1000 in this work.

To generate the training and testing data, we have to evaluate the true PDE solution u over the corresponding grids. We compute this solution to an accuracy of 10 -7 using the numerical solver BURG-ERS_SOLUTION , which is implemented in [14]. This is available from https://people.math.sc. edu/Burkardt/m_src/burgers_solution/burgers_solution.html under the GNU LGPL license.

Plots. As in Appendix B.5, our error plots display the geometric mean error value and a shaded region corresponding to one (geometric) standard deviation. We also plot the points generated by the final iteration of CAS. We visualize the density of these points using the package densityScatterChart in MATLAB using 'ksdensity' as density method and α ∈ [0 . 2 , 0 . 5] . See https://www.mathworks.com/matlabcentral/fileexchange/ 95828-densityscatterchart?s_tid=FX_rc3_behav for more details.

0.75

0.7

0.65

0.8

0.8

0.7

0.6

0.6

0.9

0.6

0.55

0.4

0.8

0.4

0.6

0.5

test error

0.45

0.2

0.7

0.2

0.5

0.6

0

0.4

-0.2

0.5

-0.2

0.4

0.35

-0.4

0.4

-0.4

0.3

0.3

-0.6

0.3

-0.6

+-ReLU 5×50-MCS

-ELU 5×50-MCS

ELU 5 × 50-CAS

-0.8

0.2

-0.8

0.2

0.25

2000

4000

m

6000

8000

10000

-1

0.2

0.4

0.6

0.8

1

0.1

-1

0

0.2

0.4

0.6

0.8

1.6

0.8

0.8

1.4

0.6

+

1.2

0.6

E(u)

0.4

0.4

1.2

test error

10-1

0.2

0.2

0

0.8

0.8

-0.2

0.4

0.6

-0.2

-0.4

""

0.6

tanh 3×30-CAS

→-tanh 3×30-MCS

sigmoid 3 × 30-CAS

-0.6

0.4

0.6

0.4

10-2

-sigmoid 3 × 30-MCS

2000

4000

6000

8000

-0.8

0.2

0.2

m

0.2

0.4

0.6

0.8

0

0.2

0.4

0.6

0.8

1

0.75

0.9

0.65

0.7

0.8

0.8

0.8

0.8

test error E(u)

0.6

0.6

0.4

0.7

0.6

0.4

0.7

0.2

0.6

0.2

0.6

0.45

-0.2

0.5

0.4

-0.2

0

0.5

0.4

-0.4

-0.4

0.4

0.35

ELU 3 × 30-MCS

ReLU 3 × 30-CAS

-→- ReLU 3 × 30-MCS

ELU 3 × 30-CAS

-0.8

-0.6

E'0

0.2

-0.6

-0.8

0.3

2000

4000

6000

8000

0

0.2

0.4

0.6

0.8

1

0.2

0.4

0.6

0.8

0.2

m

Implementation, hardware and compute time. Our experiments have been implemented in TensorFlow version 2.5. This implementation has been inspired by works [3, 19].

The computations were performed in single precision using NVDIA Tesla P100 GPUs on Compute Canada's Cedar compute cluster. To be precise, each node used in these experiments has 24 CPU cores and 128GB of memory and four Tesla P100 GPUs with 16GB memory. See https://docs. alliancecan.ca/wiki/Cedar for further information.

The compute time is roughly one day for the computations involving the ρ 3 × 30 and ρ 5 × 50 DNNs and 2 days for those involving the ρ 10 × 100 DNNs. Therefore, the total compute time is approximately four days.

10°

0.8

2.5

0.8

0.6

0.6

test error E(u)

0.4

0.4

0.8

10-1

0.2

0.2

0

1.5

0

0.6

-0.2

0.2

-0.4

-0.4

0.4

102

→tanh 10 × 100-CAS

-tanh 10 × 100-MCS

-0.6

-0.6

-sigmoid 10 × 100-MCS

→sigmoid 10 × 100-CAS

-0.8

0.5

-0.8

0.2

2000

4000

6000

8000

10000

m

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

0.7

0.8

1.2

0.8

0.6

0.6

0.6

0.9

(n)

0.5

0.4

0.4

0.8

test error

0.4

0.2

0.8

0.2

0.7

0.6

0

-0.2

0.6

-0.2

0.5

0.3

-0.4

-0.4

0.4

→ReLU 10 × 100-CAS

*-ReLU 10 × 100-MCS

-0.6

0.4

-0.6

0.3

ELU 10 × 100-CAS

-ELU 10 × 100-MCS

-0.8

0.2

-0.8

0.2

0.2

2000

4000

6000

8000

10000

0.2

0.4

0.6

0.8

1

0.2

0.4

0.6

0.8

m

D.6 Further experiments and discussion

In Fig. 9 we show the same results as in Fig. 3, except using the ReLU and ELU activation functions instead. Here, none of the methods lead to a decreasing test error as the number of samples increases, although the errors for MCS are somewhat larger than those of CAS. As is evident from the plots of the samples points, CAS fails to adapt the sampling measures to the PDE solution in this experiment, unlike in the case of Fig. 3.

This experiment indicates that the performance of CAS in relation to MCS is heavily influenced by the choice of architecture. To further explore this phenomenon, in Fig. 10 and Fig. 11 we consider 3 × 30 and 10 × 100 DNNs, respectively. In the former case, we once more see no decrease in the test error for the ReLU and ELU activation functions. The test error for the tanh and sigmoid activation functions does decrease with increasing m . Yet, CAS leads only to a marginal benefit over MCS for the tanh activation, and leads to worse performance for the sigmoid activation. As can be seen from the plot of the points, the CAS fails to adequately cluster the points in the case of the latter.

In Fig. 11, we consider 10 × 100 DNNs and see decreasing errors for the tanh and sigmoid activations, with CAS giving better performance than MCS. For the ELU activation function, while the error is (slowly) decreasing, CAS actually performs worse than MCS. Once more, for the ReLU activation function, we see no convergence.

The discrepancy in performance between the different activation functions is mirrored when we examine the training loss. In Fig. 12 we show the training loss versus number of epochs. For the best-performing tanh and sigmoid activations, we see significant decrease of the training loss, whereas for the other activations we see slower or sometimes no decrease. Note that all DNNs are trained using the same hyperparameters. This behaviour is in some senses unsurprising, since the worst-performing activation functions are only piecewise smooth and the training loss involves the PDE operator. Another notable feature is that MCS generally attains a smaller training error than CAS. Therefore, there is scope to further improve the performance of CAS by improving the training procedure.

102

102

tanh 5 × 50-CAS

tanh 5 × 50-MCS

100

sigmoid 5×50-MCS

sigmoid 5 × 50-CAS

101

100

10-2

10-1

104

10-2

10-6

10-3

ReLU 5 ×50-CAS

ReLU 5×50-MCS

ELU 5 × 50-CAS

ELU 5 × 50-MCS

10-8

0.5

1

1.5

2

2.5

10-4

0

0.5

1

1.5

2

2.5

×105

×105

102

102

ReLU 3×30-CAS

ReLU 3×30-MCS

ELU 3×30-CAS

100

ELU 3 ×30-MCS

101

10-2

10°

10-4

10-1

10-6

tanh 3 × 30-CAS

tanh 3 × 30-MCS

sigmoid 3×30-CAS

sigmoid 3 × 30-MCS

10-8

0

0.5

1

1.5

2

2.5

10-2

0

0.5

1

1.5

2

2.5

×105

×105

104

104

ReLU 10 × 100-CAS

102

sigmoid 10 ×100-CAS

tanh 10 × 100-MCS

sigmoid 10 × 100-MCS

103

ELU 10× 100-MCS

ReLU 10 × 100-MCS

ELU 10 × 100-CAS

102

100

101

10-2

100

10-1

10-4

10-2

10-6

10-3

10-8

0

0.5

1

1.5

2

2.5

0

0.5

1

1.5

2

2.5

×105

×105

Finally, in Fig. 13 we plot the functions ˜ K c , c = 1 , 2 , 3 , that are used to generate the sample points at the final iteration (see Appendix D.4). Here, once again we see that the tanh and sigmoid activation functions lead to functions ˜ K c that are peaked according to the behaviour of the true PDE solution u ∗ . However, the ReLU and ELU activation functions fail to capture this behaviour.

E Proofs of the theoretical results

In this appendix, we give the proofs of the results from the main paper. We commence with the following lemma, which gives several properties of the generalized Christoffel function that will be needed later.

Lemma E.1 (Properties of K ( F )( θ ) ) . The generalized Christoffel function K (see Definition 2.6) has the following properties.

-1

-0.5

0

0.5

1

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0.022

0

0.2

0.4

0.6

0.8

1

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

-1

-0.5

0

0.5

1

0

0.005

0.01

0.015

0.02

0.025

0.03

0

0.2

0.4

0.6

0.8

1

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

-1

-0.5

0

0.5

1

0

0.02

0.04

0.06

0.08

0.1

0.12

0

0.2

0.4

0.6

0.8

1

0

0.01

0.02

0.03

0.04

0.05

0.06

-1

-0.5

0

0.5

1

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0

0.2

0.4

0.6

0.8

1

0

0.005

0.01

0.015

0.02

0.025

1

0.8

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

-1

0

0.2

0.4

0.6

0.8

1

1

0.8

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

-1

0

0.2

0.4

0.6

0.8

1

0.8

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

-1

0

0.2

0.4

0.6

0.8

0.8

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

-1

0

0.2

0.4

0.6

0.8

(i) Let F be a finite-dimensional subspace of X . Then

for any orthonormal basis { f i } n i =1 of F . Furthermore, if Y has dimension p then

In particular, (E.1) holds with equality when dim( Y ) = 1 .

(ii) Let F 1 , F 2 ⊆ X . Then

(iii) Let F 1 , F 2 be finite-dimensional subspaces of X . Then

̸

Proof. We commence with the proof of (i). Write f ∈ F , f = 0 , as f = ∑ n i =1 c i f i for c i ∈ C with ∑ n i =1 | c i | 2 = ∥ f ∥ 2 X . Then

and therefore

For the lower bound, let { e 1 , . . . , e p } be an orthonormal basis of Y . Fix j ∈ { 1 , . . . , p } and consider f = ∑ n i =1 c i f i , where c i = ⟨ e j , L ( θ )( f i ) ⟩ Y . Then

We also have

Therefore, we deduce that

Since j was arbitrary, we now sum over j and use Parseval's identity to obtain

as required.

Part (ii) follows immediately from the definition of K . Now consider (iii). Write F = F 1 + F 2 = G 1 ⊕ G 2 , where G 1 = F 1 and G 2 = F 2 ∩ F ⊥ 1 satisfies G 2 ⊥ G 1 . Let f ∈ F , f = 0 , be arbitrary and write f = g 1 + g 2 for g i ∈ G i , i = 1 , 2 . Then

where in the last step we used the fact that G 2 ⊆ F 2 . Since f was arbitrary, the result now follows. Finally, consider (iv). By the definition of K , B ( F ) and linearity,

as required.

̸

For the remaining results it is useful to introduce some additional notation. Let

be the overall measurement space . Note that this is a Hilbert space, when equipped with suitable norm. If y = ( y ic ) is an arbitrary element of Y , we take this norm to be

Now let m = m 1 + . . . + m C , D = ⊗ C c =1 ⊗ m c i =1 D c be the tensor-product space and write ( θ ic ) for an arbitrary element of D , where θ ic ∈ D c for i = 1 , . . . , m c and c = 1 , . . . , C . Then we define the overall sampling operator

Note that the operator on the right-hand side is given by

Further, we note that the nondegeneracy condition is equivalent to

Here τ = ⊗ C c =1 ⊗ m c i =1 µ c is the probability measure of the full sample (Θ ic ) := Θ .

Next, given measurements as in (2.1) or (4.4), we define the scaled measurements ¯ y = (¯ y ic ) ∈ Y by

Then, with M as defined in E.2, the regression problem (2.2) is equivalent to

With this in hand, we now prove Theorem 4.2. For this, it is first useful to prove a special case of Theorem 4.2 when F is itself a subspace.

Lemma E.2 (Special case of Theorem 4.2 for subspaces) . Consider the setup of Theorem 4.2, where F is a subspace of dimension at most n . Suppose that

for some 0 < γ < 1 , where

Then (2.5) holds with probability at least 1 -δ .

Proof. The proof of the result in this case is a straightforward application of the matrix Chernoff bound [115]. Let { f i } n i =1 be an orthonormal basis for F . Write an arbitrary f ∈ F as f = ∑ n i =1 c i f i , so that ∥ f ∥ X = ∥ c ∥ ℓ 2 , where c = ( c i ) n i =1 ∈ C n . Then

where X ic are the independent, self-adjoint, n × n random matrices with

Thus, the result is equivalent to the condition

Notice that X ic ⪰ 0 and, due to (E.3),

Now, with f = ∑ n i =1 c i f i ∈ F , we have

Thus,

almost surely. The result now follows from the matrix Chernoff bound.

Proof of Theorem 4.2. Let f ∈ B ( F -F ) . Then there is j ∈ { 1 , . . . , d } and f j ∈ F j ∩ B ( F -F ) such that | | | f -f j | | | ≤ t , where t = √ αϵ/ 2 . Observe that

∥

Now, with probability one,

Using this and the fact that β ≥ α , we deduce that

Now let E be the event

and note that our goal is to show that P ( E ) ≥ 1 -δ . By linearity, E is equivalent to

Hence, by the above argument, we have

where E j is the event

By the union bound

Suppose first that n ≥ 1 . Then K ( F j )( θ ) ≤ K ( ˜ F )( θ ) and the condition (4.1) on the m c and Lemma E.2 give that P ( E c j ) ≤ δ/d .

The gives the result for n ≥ 1 .

For n = 0 , we have that F j = { f i } is a singleton. Notice that F j ⊆ span( F j ) and K c ( F j )( θ ) = K c (span( F j ))( θ ) . Hence we may apply Lemma E.2 with n = 1 . We deduce once more that P ( E c j ) ≤ δ/d . This gives the result for n = 0 as well.

Remark E.3 (The norm in Theorem 4.2) Write F ′ = B ( F -F ) . Then it follows from the definition of the K c that

Therefore, the cover in Theorem 4.2 could be constructed with respect to the X -norm, up to a change in the constant t . Notice that the quantity that bounds | | | f 1 -f 2 | | | in terms of ∥ f 1 -f 2 ∥ X involves the generalized Christoffel functions K c ( F ′ -F ′ ) and weight functions ν c . Therefore, sampling according to the generalized Christoffel function (on a slightly modified set) also minimizes this constant, and therefore the size d of the cover in Theorem 4.2.

Proof of Corollary 4.3. If F -F = ∪ d i =1 F i is a union of d subspaces of dimension at most n , then clearly

It follows that { F i } d i =1 provides a ( | | |·| | | , n, t ) -subspace covering of B ( F -F ) for any t ≥ 0 . Hence we may apply Theorem 4.2 with this cover. Since ˜ F = ∪ d i =1 F i = F -F in this case, the result now follows immediately.

Proof of Corollary 4.4. When n = 0 , the sets F j = { f j } are singletons and we must have F j ⊆ B ( F -F ) by definition of the covering. Hence ˜ F ⊆ B ( F -F ) , which gives that K c ( ˜ F )( θ ) ≤ K c ( B ( F -F ))( θ ) . Using Lemma E.1, we deduce that K c ( ˜ F )( θ ) ≤ K c ( F -F )( θ ) . This implies the result.

Proof of Lemma 4.6. Since K ( F ) is integrable, the quantity κ ( F ) is well defined. Also, K ( F )( θ ) > 0 almost everywhere, due to the assumption on L and F . In particular, κ ( F ) > 0 . It follows that ν ⋆ is measurable, positive almost everywhere and integrable with ∫ D ν ⋆ ( θ ) d ρ ( θ ) = 1 . Clearly,

We now show that this is the optimal value. Let ν be any other such function. Then

This gives the result.

Proof of Corollary 4.7. We first prove the result when d = 1 . Let { f i } n i =1 be an orthonormal basis for F 1 = F -F . Then Lemma E.1 gives that

Therefore,

Assumption 2.4 and orthonormality now imply that

Now suppose that d > 1 . Lemma E.1 gives that

and therefore

Using the result for d = 1 , we get as required.

Remark E.4 (Lower bound on ∑ C c =1 κ c ( F -F ) ) Suppose that dim( Y c ) ≤ p ∈ N , ∀ c . Then Lemma E.1 gives, for a subspace G with an orthonormal basis { g i } n i =1 ,

Using this and following the steps of the above proof, we deduce the lower bound

Finally, we prove Theorem 4.8. For this, we first require the following lemma.

Lemma E.5. Suppose that a realization θ = ( θ ic ) ∈ D of the Θ ic gives that

for some α ′ > 0 . Then any γ -minimizer ˆ f of (2.2) satisfies

Proof. Let f ∈ F . We write

We now use the fact that ˆ f is a γ -minimizer and the fact that ¯ y = M ( θ )( f ∗ ) + ¯ e to get

as required.

Proof of Theorem 4.8. Let E be the event

Due Theorem 4.2 and the given assumptions, we have that P ( E c ) ≤ δ . Now recall that τ is the probability distribution of the full draw Θ . Then

In the first integral, we use the fact that T σ is a contraction to get

In the second integral, we use the fact that ∥T σ ( g ) ∥ X ≤ σ to get

Therefore, we obtain

Now, for the first term, we fix f ∈ F and apply Lemma E.5. Since α ′ ≥ (1 -ϵ ) α whenever E holds, we get

For the second term, we use Assumption 2.4 and (E.3) to get

and therefore

Combining this with (E.8) and recalling that f was arbitrary now gives the result.

Remark E.6 As noted, the noise bound in Theorem 4.8 assumes ρ c ( D c ) < ∞ . If this fails, one may also bound the noise term by assuming the noise takes the form

for some g ∈ X with ∥ g ∥ X ≪ 1 . In this case, one has

Hence Theorem 4.8 holds with N = β ∥ g ∥ 2 X . For other ways to estimate the noise term under different noise models, see, e.g., [93].

Proof of Theorem 2.7. Part (i) follows from Theorem 4.2, Corollary 4.3 and Lemma 4.6 with the (arbitrary) value ϵ = 1 / 2 . Part (ii) follows from part (i) and Theorem 4.8. Finally, part (iii) follows from Corollary 4.7.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an extremely general notion of active learning in the $\\ell_2$ norm. Suppose we are able to actively observe data in many different modalities. \"Active Learning\" means we can choose where we observe data in a domain, and \"different modalities\" means there are fundamentally different ways to observe the data. For example, an MRI machine can have many different coils in it, and for each coil we can measure data at a user-specified frequency. The fact that there are different coils with different physical properties corresponds to having \"different modalities\". The fact that we can choose which frequencies to measure data at corresponds to \"active learning\".\n\nThe paper studies an extremely general formalization of active learning in the $\\ell_2$ norm with different modalities in active learning. Specifically, it suggests computing a \"Generalized Christoffel Function\" (i.e. leverage score function) for each modality, and to sample from each modality at random places, chosen with probability proportional to the generalized Christoffel function.\n\nThe paper allows for an exceptionally general notion of \"sampling\", where a sample could entail something classical in numerical linear algebra, like observing a row from a matrix. It can also mean something broader, like observing the gradient of a function. Also, while the error in each modality is measured with respect to the $\\ell_2$ norm, it is the weighted $\\ell_2$ norm with respect to a given measure $\\rho$ for each modality. So, the user can choose to have (e.g.) a uniform error metric on a small interval, or to instead have a Gaussian-weighted error measure on the real line.\n\nLastly, the framework also has enough flexibility to give guarantees in some nonlinear spaces, with the main example being the union-of-subspaces.\n\nTheoretical guarantees on sample complexity are given, roughly matching the rates that would appear in the special case of leverage score sampling for matrices. Further, many experiments and concrete instantiations of the general framework are presented. Applications vary from approximating function with multivariate polynomials, to optimizing MRI reconstructions, to solving PDEs with physically informed neural nets.\n\nSTRENGTHS:\nThe paper is also 40 pages of really intense serious research. The appendix is chalk full of interesting special case studies, demonstrating the flexibility of their theory and the practical algorithms it suggests. The paper is so big and full of ideas that the NeurIPS peer review system, where I don't have enough time to read it cover-to-cover, really does this paper a disservice. I'd love to have 2 months to focus on this paper and understand the ins-and-outs of every case study and theorem. However, I don't have that time, so I'll presume the correctness of everything in the appendix and strongly recommend publishing this paper.\n\nThe paper is exceptionally general, doing a great job of generalizing much of the recent research on leverage score sampling beyond matrices. Lots of existing work has slowly generalized one or two ideas from leverage score sampling to Hilbert spaces or non-linearity or whatever else fits that work's setting. This paper really goes the extra mile and writes out a ridiculously general setting that covers:\n- Fitting a polynomial to a function, where we can observe both the function and it's gradient\n- Fitting a polynomial to a function, where we can choose if we want to observe the function or it's gradient at each query (in case one is more expensive than the other, we can consider them separate modalities with separate sample complexities).\n- Iteratively refined approximations to solving a PDE\n- Reconstructing MRI data from observations at carefully chosen frequencies\n\nThe general framework is phrased in terms of normed spaces, Hilbert spaces, measurable functions, and other abstract math that really gives the user freedom to do what the want. The theorems also apply to this extremely general framework, so the user immediately has Christoffel sampling guarantees with a likely near-optimal sample complexity out-of-the-box.\n\nI think this paper is original in it's powerful results in extreme generality, and is general in it's applications. The paper's results are high quality sample complexity guarantees. The paper is broadly well written (but this is a bit of a weaker point). The paper should be significant in its expressive power owing to its applicability to many domains.\n\nI strongly recommend publishing it.\n\nWEAKNESSES:\nThe paper does suffer a bit on the clarity front. I'm familiar with leverage score sampling (and even some infinite-dimensional generalizations thereof), so I'd think that a generalized Christoffel function framework would be easy enough to understand. This was not the case though, and the extremely general framework was daunting an unapproachable to read about.\n\nThere's even an appendix subsection devoted to relating the extremely general framework back to the leverage score form I'm comfortable with. However, this description isn't super clear and even has (as far as I can tell) several minor errors which really harm legibility.\n\nThe general framework (section 2.1) is just a list of math definitions stated without any running example of what each mathematical object is. It's very unintuitive what the difference between a \"measurement domain\" and a \"measurement space\" are [Lines 85-87]. I'd strongly recommend the authors decide on a running example to present alongside all of the math definitions, which would make such definitions more intuitively distinct and clear.\n\nThe experiment and case study examples (section 3) are also pretty hard to approach. Section 3 consistently summarizes ideas at an extremely high level, and pushes the rigorous instantiations of the general framework to the appendix. While this makes sense from a page-limit perspective, it does harm the clarity and reading experience. Unfortunately, under the constraints of the NeurIPS format, I'm not sure a better option is available. Perhaps the authors can avoid this extreme appendicizing in the arXiv version of their paper?\n\nEven when reading the appendices in good detail, there's bits and pieces that I don't follow (especially in the MRI example). Everything I understood was clearly correct, but sometimes the technical domain-specific language gets a bit overwhelming making the examples hard to follow.\n\nThis clarity point is not a game-breaking issue, but another pass at making the technical material more approachable would benefit the paper, especially when it's targeting such a broad audience of both theoretical and empirical data scientists.\n\n\n_As a side note, it's curious to note one particular limitation of the framework -- it cannot explicitly handle ridge regression and ridge leverage scores, which is essential for some prior work on infinite dimensional leverage score sampling, like paper [12] in the references. I don't see any reason why CS4ML couldn't handle ridge regression / ridge leverage scores though._\n\nQUESTIONS:\nFirst, I have some moderately technical qualms with the paper:\n1. In Figure 1, if I understand Appendix B.6 correctly, is it correct that MCS (monte carlo sampling / non-christoffel sampling) is performing worse with more samples because of numerical instability / poor conditioning? Or, like [Line 194] says, is it because the sample complexity of MCS sampling is so bad that the error.... _gets worse_ as the sample complexity grows?\n1. The Appendix doesn't seem to have a fully accurate and consistent instantiation of leverage scores into the CS4ML framework. Specifically, $\\mathbb{U} = \\mathbb{C}$ on [Line 672] seems at odds with $\\mathbb{U} \\subseteq \\mathbb{X}$ when $\\mathbb{X} = L\\_\\rho^2(D)$. I'm pretty sure that $\\mathbb{U} = \\text{span}(A)$ in leverage score sampling? Also, it feels pretty inconsistent to describe $\\mathbb{V}$ as a subset of $\\mathbb{R}^N$ but to describe $\\mathbb{X}$ as a weighted $L_2$ function space on $[n]$. Understanding that much is already non-obvious from the notation, but would be much clearer if you just wrote $\\mathbb{X}=(\\mathbb{R}^N, \\|\\|\\cdot\\|\\|\\_{\\mathbb{X}})$ where $\\|\\|\\vec v\\|\\|\\_\\mathbb{X}^2 = \\frac1N \\|\\|\\vec v\\|\\|_2^2$. Also, there's no reason (afaik) for $\\mathbb{Y}$ to be complex instead of real. Further, the sampling function should more explicitly be about looking up an item from a vector, something like $L(y)(\\vec u) = \\sqrt{p(y)}u(y) = \\frac{1}{\\sqrt N} [u]\\_{y}$. Lastly, from my algebra, I found that $\\alpha=\\beta=\\frac1N$ in equation (2.4) for the leverage score setting. Did I make an error, or is there an adjustment to the instantiation that can instead make $\\alpha=\\beta=1$?\n\n---\n\nAfter that, I've got smaller qualms / typos / recommended edits:\n1. [Line 92] The choice to make $(D_c,\\mathcal{D}_c)$ makes for two symbols which look identical at a first (and second) glance. I'd change at least one of these symbols so it's clear that the first and second items in the pair are different.\n1. [Line 116] Add parenthesis around 2.1\n1. [Line 142] Remove $K=$\n1. [Line 156] Why isn't there rescaling by $\\sqrt{p(y)}$ here, like there was in the leverage score appendix's definition of $L$?\n1. [Figure 1] Explain the confidence intervals in the caption\n1. [Line 168] The placement of \"See Section 4.3\" is not great, since Section 4.3 is about translating an \"in-probability bound\" into an \"expectation bound\". I'd move \"See Section 4.3\" one sentence earlier.\n1. [Line 226] Redefine PINN here. I know it's defined in the abstract, but I'd do it again here.\n1. [Figure 3] Add titles to the plots\n1. [Lines 230-239] I completely failed to understand what this paragraph is trying to say.... I just don't understand the problem or how it's being solved.\n1. [Section 4] Swap $\\varepsilon$ and $\\delta$. I'd **very heavily** recommend this. The choice of $\\delta$ to mean failure probability and $\\varepsilon$ to mean accuracy is extremely standard. I would not mess with this; it's very confusing to read.\n1. [Line 269] Interpret this big messy norm $\\|\\|\\|\\cdot\\|\\|\\|$ somewhere. It looks scary, but I don't think it really should be that scary?\n1. [Line 269] Missing $c$ subscript on $\\nu$.\n1. [Equation 4.1] Explore this sampling complexity. What is it really saying, comparing our sampling distribution to the Christoffel function? Can we use it to get a sorta \"coherence assumption\" for uniform sampling / MCS? This should be somewhat intuitive, but it really stands to be formally explored a bit more in the writing.\n1. [Lines 271-272] Why is it \"particularly relevant\"? Why's it hard to understand the Christoffel function of the cover but easy to understand the Christoffel function of $\\mathbb{U}$. I sorta intuitively believe it's true, but you should justify it.\n1. [Line 311] Consider calling $\\mathcal{T}_\\theta$ a \"shrinking operator\" maybe? or a \"regularization operator\" or something? It's not very truncation-like imo, but it's really a small preference. This is a very mild/soft recommendation.\n1. [Lines 316-318] Please formalize this added-penalty approach. It's the generalization of ridge regression, which feels very worth exploring.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes a general framework for active learning. It claims that the proposed method can help achieve near-optimal sample complexity. In additional to the theoretical results, this paper also present 3 use cases, showing favorable results for the proposed method.

STRENGTHS:
Important topic with claimed good results.

WEAKNESSES:
1. Poor presentation: Very heavy theoretical setup buried the connection to solving any real active learning problem. No high-level description of the algorithm and its advantages over the existing active learning algorithms. No insights are provided based on the theoretical derivation.
2. Poor experiment comparison: The proposed method was only compared with random sampling. There's no comparison with any existing active learning method.

QUESTIONS:
1. line 19, p. 1: In ML literature, features are represented by $x_i$ instead of $y_i$ while the latter denotes the label, which is $f(y_i)$ in your case.
2. line 38, p. 2: In your example of computational imaging, is one image an example or one pixel an example? Both cases can occur in real applications.
3. line 60, p. 2: Do you mean the label function $f$ is in a Hilbert space? For any function with a set of real-valued parameters (e.g., a neural network), if it's represented by a vector of its real-valued parameters, is such a function in a Hilbert space?
4. line 61, p. 2: What does "data" in "data arises" mean? Examples of features and label pair?
5. line 179, p. 5: You claim "there are many settings where one can acquire both function values and values of its gradient." Can you give some real examples of such settings?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework for active learning (here meaning that the user controls the sampling strategy according to which the locations of the measurements are made) designed to handle various cases of regression (vector-valued, multimodal, and more).\n\nTo do so, it introduces the concept of generalized Christoffel functions, that will drive the choice of sampling strategies.\nA statistical analysis of the risk associated to the empirical risk minimizer associated to the square loss is conducted, showing that a log-linear number of observations (with respect to quantities related to the geometry of the hypothesis space) allows for efficient training using the Christoffel sampling strategy.\n\nNumerical experiments on three different problems (all cast inside the cs4ml framework) complement the approach, showing improvements over the naive Monte-Carlo sampling.\n\nSTRENGTHS:\n- The proposed approach is novel and encompasses many valuable problems to the machine learning community.\n- Mathematical details are sound.\n- Experiments show clear improvements.\n- The appendix contains a lot of remarks, additional experiments, that further develop the framework.\n- I share the author's enthusiasm regarding the potential of the framework, which is far from explored.\n\nWEAKNESSES:\n- It sometimes feels that the paper is overly complicated for the sake of generality, without benefits (see questions).\n\n- While fairly general, the framework assumes that the user has access to the data generation strategy (not the targets, but the inputs) which is often not the case in practice.\n\n- Overall, the paper is very dense and notation heavy. It is impossible to read it without a constant back and forth to the (31 pages long) appendix, which makes the reviewer wonder why the 9 pages limit exists for this conference. In my opinion, it would be beneficial to have a longer (i.e. journal) version of the paper, as the supplementary contains many interesting remarks that help a lot to the understanding of the ideas.\n\nQUESTIONS:\nI found the notation to be quite heavy and not intuitive at all. I suggest to change the name convention adopted here - while all notation choices are arbitrary, they need to be driven by simplicity and logic. In particular:\n\n- I suggest adopting $x_{ic}$ for the measurement locations, to be conform to the classical $y = f(x) + \\epsilon$ regression statistical model. Anything evoking something we usually put inside a function call or put measures on is ok ($\\theta_{ic}$ ?).\n- It feels weird not to have the random variable $Y_c$ living in the space $\\mathbb{Y}$. Measurements could be $y_{ic}$ instead of $b_{ic}$.\n- To lower the number of different letters, the true function associated to the generating process could be $f^\\star$, and the candidates could be $f$ (instead of $u$), keeping the estimate $\\hat{f}$.\n- In the regression literature, hypothesis spaces are usually referred to as $H$ or $F$. If the measurements locations are $x$, the name of the object space has to change as well.\n- In definition 2.4, you introduce yet another variable name $\\mathbb{V}$ that could just be the variable name of the hypothesis space, to improve clarity.\n\nOther questions/comments:\n\n- Where do you use non-boundedness of the measurement operators ? Could you not use the standard notation that elements of $\\mathcal{L}(\\mathbb{X}, \\mathbb{Y}_c)$ are bounded linear operators ? Usually unbounded operators are not defined on the whole space $\\mathbb{X}$ but only on their domain.\n\n- Could you comment on the need to parameterize the measures using two other measures ? $\\mathrm{d}\\mu_c(y) = \\nu_c(y) \\mathrm{d}\\rho_cc(y)$ ? In the end, we first choose $\\rho$ then take $\\nu$ according to the CS strategy, so what is the influence different choices of $(\\rho, \\nu)$ yielding the same $\\mu$ ?\n\n- Empirical nondegeneracy as presented in Eq. 2.5 is a void concept, unless $\\alpha' \\approx \\alpha$ and $\\beta' \\approx \\beta$ are made precise.\n\n- In Theorem 2.5, the symbol you use for the estimator is not defined until way too long after.\n\n- The notion of optimal sample complexity is not well explained enough and I struggle to see exactly with respect to what it is optimal. How can it be optimal given the crude estimate for the sum of Christoffel functions ?\n\n- Section 3.1: Have you studied the impact of the choice of $\\rho$ on numerical performances ? What if $\\rho$ were a laplace measure ? How do you choose $f$ in the experiments ? Also, why not write $(y_i, f(y_i), \\nabla f (y_i))$ ? This avoids the supercharge of the subscript notation in line 752.\n\n- Proposition 4.3, 4.4 could be renamed corollaries or examples.\n\n\nTypos:\n\n- Line 61: vector-valued instead of vector space valued\n- In assumption 2.2, the mappings $L_c$ are from $D_c$ to $\\mathbb{Y}_c$. (also in definition 2.4, check the whole text)\n- In Eq. (2.2) and (2.3) it should be $\\forall u \\in \\mathbb{U}$.\n- \"Hence, the sampling operators and measures preserve the X-norm in expectation\" is very vague.\n- Appendix B1: line 759: no need for the complex conjugate. Why do you need to work in C after having defined your function as R-valued ? Line 763 rho is the same measure for all indexes k.\n\nLIMITATIONS:\nMost limitations are discussed in the paper. I would also like to mention that the cs4ml framework is not suited to all regression problems, but only to those where one controls the sampling strategy. Alas, in practice you seldom know $\\rho$, because the data distribution is often unknown. (Correct me if I'm wrong)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

comments: SUMMARY:
The authors introduce an active learning framework for regression problems based on the concept of generalized Christoffel functions. The proposed approach is applicable to a broad range scenarios and it is evaluated on several scientific computing tasks.

STRENGTHS:
The manuscript present a comprehensive explanation over the main ideas behind the proposed CS4ML framework. The main motivation is relevant (active learning for diverse data scenarios) and the authors point promising directions for further investigations.

The work is well organized, which is especially important for theory focused contributions. I also praise the detailed provided supplementary material.

WEAKNESSES:
The experimental section, although tackling diverse tasks, unfortunately compares the proposal only to an inactive learning strategy (Monte Carlo sampling), which diminishes the relative impact of the introduced method. I appreciate this point being highlighted as a limitation by the authors, but it does leave the reader wanting a bit more.

QUESTIONS:
Although the manuscript lists 118 references, I think it lacks a more thorough in-text discussion with other available active learning strategies. Which literature gaps are addressed? Which scenarios are not entirely covered by standard methods? 

As a final minor observation, the left plot in Fig. 2 present some difficult to distinct line patterns which could be improved.

LIMITATIONS:
The manuscript states sufficiently its limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-15