On the Benefits of Active Data Collection in Operator Learning
Authors: Unique Subedi
Ambuj Tewari
Venue: ICML 2025 spotlightposter (2025)
Keywords: Operator Learning
Active Learning
PDEs
Primary area: theory->active_learning_and_interactive_learning
TL;DR: We study active data collection strategies for operator learning and establish their provable advantage over passive sampling approaches.

Abstract: We study active data collection strategies for operator learning when the target operator is linear and the input functions are drawn from a mean-zero stochastic process with continuous covariance kernels. With an active data collection strategy, we establish an error convergence rate in terms of the decay rate of the eigenvalues of the covariance kernel.  We can achieve arbitrarily fast error convergence rates with sufficiently rapid eigenvalue decay of the covariance kernels. This contrasts with the
passive (i.i.d.) data collection strategies, where the convergence rate is never faster than linear decay ($\sim n^{-1}$). In fact, for our setting, we show a \emph{non-vanishing} lower bound for any passive data collection strategy, regardless of the eigenvalues decay rate of the covariance kernel. Overall, our results show the benefit of active data collection strategies in operator learning over their passive counterparts.

On the Benefits of Active Data Collection in Operator Learning
Unique Subedi' Ambuj Tewari
Abstract
F under suitable metric. The hope is that for new in- put function f. evaluating ū = (f) will be considerably faster than solving for u through traditional methods, with minimal loss in accuracy.
We study active data collection strategies for oper- ator learning when the target operator linear and the input functions are drawn from a mean-zero stochastic process with continuous covariance ker- nels. With an active data collection strategy, we establish an error convergence rate in terms of the decay rate of the eigenvalues of the covariance kernel. We can achieve arbitrarily fast error con- vergence rates with sufficiently rapid eigenvalue decay of the covariance kernels. This contrasts with the passive (i.i.d.) data collection strategies, where the convergence rate is never faster than linear decay (~ n='). In fact, for our setting, we show a non-vanishing lower bound for any passive data collection strategy, regardless of the eigenvalues decay rate of the covariance kernel. Overall, our results show the benefit of active data collection strategies in operator learning over their passive counterparts.
In this work, we study the sample complexity of operator learning. Specifically, given an operator F, how many input- output pairs
are necessary to estimate an operator F. such that Fa(f) = F(f) for all relevant f? This question has been studied in several specific contexts, such as for linear operators with fixed singular value de- composition by de Hoop et al. (2023) and Subedi & Tewari (2024), for Lipschitz operators by Liu et al. (2024), and for the random feature model by Nelsen Stuart (2021). These are just few representative works, and we refer the reader to (Kovachki et al., 2024b, Section 5) for more comprehensive review of such sample complexity results.
A common theme of these sample complexity analyses is that they are conducted within the framework of traditional statistical learning (Kovachki et al., 2023, Section 2.2). In the statistical setting, the learner has access to training sam- ples {(ʃ},, ((E))) where ʃj ~idd l from some proba- bility measure μl, and the objective is produce estimator F, such that (f) = F(f) on average over test samples f ~ This scenario is also referred to as the passive learn- ing setting. Under reasonable non-trivial assumptions, the best achievable rate of error convergence in this setting is ~ 1 1/n, when F, is evaluated under an appropriate metric, say the p-th power of the - 2.-Bochner norm.
1. Introduction
There is an increasing interest in using data-driven methods to estimate solution operators of partial differential equa- tions (PDEs) encountered in scientific applications. To set up the problem, consider X Rd and linear PDE of the form Lu f. subject to the boundary condition u(x) 0 for all boundary The goal, given function is to find the corresponding solution that satisfies the PDE.
1.1. Our Contribution
Traditionally, numerical PDE solvers are used to compute u from f. In contrast, operator learning focuses on approx- imating the solution operator of the PDE (Lu et al., 2021; Kovachki et al., 2023). Under typical conditions, this PDE has linear solution operator F such that u F(J) for all in an appropriate function space. Given set of training samples (f1, v.)..... (Jn, u,), the aim is to learn an opera- tor Fr, that closely approximates the true solution operator
In this work, we go beyond the passive statistical setting and study operator learning where the learner is not restricted to iid samples from source distribution and can use active data collection strategies. We focus on the case where the operator of interest F is a bounded linear operator and is a distribution with zero mean and the covariance structure defined by continuous kernel K. Such distributions in- clude Gaussian processes with common covariance kernels. For a given covariance kernel K, our main result provides active data collection strategy, estimation rule, and establishes an error bound for the proposed estimator in terms of the eigenvalue decay of the integral operator of K. Formally, if A 2 Xg 2 are the eigenvalues of the
Department of Statistics, University of Michigan, Ann Arbor, USA. Correspondence to: Unique Subedi <subedi @ umich.edu>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Benefits of Active Data Collection in Operator Learning
integral operator of K, there exists an active data collection strategy and estimation rule such that the estimator obtained using actively collected input-output pairs achieves the following error bound:
strategies. However, the upper bound in (Kovachki et al., 2024a, Theorem 3.3) is derived assuming that input func- tions UI...., U/ are drawn i..dd from μ.. Their proof, based on standard empirical risk minimization (ERM) analysis, achieves a convergence rate that, at best, matches the Monte Carlo rate of n 1/22 Moreover, their lower bounds apply to both active and passive data collection strategies, suggesting that, for the nonparametric operator classes considered by Kovachki al. (2024a), active learning provides clear advantage over passive approaches. Exploring whether an adaptive data collection strategy, informed by the covari- ance of μ and targeting smaller subclasses within these broad nonparametric classes, could yield faster convergence rates remains an interesting direction for future research.
+
2 op
A
The e² captures the error of approximation oracle O for F that the learner has access to. For example, O could be the PDE solver used generate training data for oper- ator learning. Generally, & > 0 is the irreducible error of the bound. The second term i A²) is a reducible error, which goes to 0 as + oo for continuous kernels K' under the bounded domain. For example, for the covari- ance operator ax- + 31) used by Li et al. (2021) and Kovachki et al. (2023), we show that the reducible error vanish at the rate X n ( Taking 2 >> d, one can achieve any polynomial rate of decay. In fact, given any rate R., + as n + oo, one can always construct a continuous kernel K such that the reducible error decays faster than Ry, Thus, arbitrarily fast rates can be obtained using active data collection strategies. Our main result is formalized in Theorem 3. and the proof is based on the celebrated Karhunen- decomposition for functions drawn from with covariance kernel K.
Additionally, Boullé et al. (2023) shares our objective of achieving faster convergence rates for PDEs with linear so- lution operators, but there are notable differences between their results and ours. First, their approach requires stronger control over the Hilbert-Schmidt norm of the operator F. whereas we only require control over the operator norm. Notably, the Hilbert-Schmidt norm can be arbitrarily larger than the operator norm. Second, their estimator uses the spe- cific structure of F. particularly the Green's is function, while we rely on black-box access to F via an E-approximate or- acle. Lastly, although both approaches introduces a term measuring the quality of training data (e in our bound and Ie in theirs), their definition of I2 is more technical and less intuitive. However, their guarantee is stronger, as their upper bound applies uniformly to any L -'-integrable input function, whereas our guarantees hold in expectation for inputs drawn from the distribution Jr.
Furthermore, in Theorem 4.2, we show that, irrespective of the decay rate of the eigenvalues of the covariance kernel K, there always exists a bounded linear operator F and distribution jr with covariance kernel K such that the minimax estimation error fails to converge to 0 under any passive (i.i.d)) data collection strategy. In particular, for every n, even when e = 0, we establish the minimax lower bound of
Our work considers the setting where μ is defined by a stochastic process with a specific covariance structure. Such a was taken to be a Gaussian process with mean zero and covariance given by a(-v² + ++)1== in (Bhattacharya et al., 2021: Li et al., 2021; Kovachki et al., 2023). The use of Karhunen- functions is also discussed by Boullé & Townsend (2023, Section 4.1). Our upper bound also share conceptual similar- ities with results in (Lanthaler et al., 2022: Lanthaler, 2023), who established approximation error bounds, rather than estimation, in terms of s eigenvalues of covariance operator.
|
op A
under any passive data collection strategy. That is, the lower bound does not vanish even as n - xx. Collectively, Theorems 3.1 and 4.2 establish a clear advantage of active data collection strategy for operator learning.
decomposition for generating input
1.2. Related Works
Recent work by Musekamp et al. (2024) considers active methods for operator learning. However, in contrast to our approach of using linearity of the solution operator and the distributional family of interest, their methods rely on estimating uncertainty and identifying coreset. Addition- ally, their study is purely empirical and lacks theoretical guarantees. In similar spirit, Li et al. (2024) study using active learning to select input functions from multiresolution datasets to lower the data cost.
Finally, we highlight the ICML 2024 tutorial by Azizzade- nesheli (2024), who mentions active data collection as an important future direction for operator learning. We also ac- knowledge the extensive literature on the learning-theorett foundations of active learning (Settles, 2009). The active learning framework we adopt is known as the membership query model, which has a rich history in learning theory (Angluin, 1988). A more detailed discussion of various active learning models within the learning theory literature is deferred to Section 3.4.
On the theoretical side, a closely related work is by Ko- vachki et al. (2024a), who allow for active data collection
2

---

On the Benefits of Active Data Collection in Operator Learning
2. Preliminaries
can now meaningfully talk about probability distribution over (22) induced by the stochastic process h.
2.1. Notation
Accordingly, given continuous covariance kernel K, let P (K)) denote the set of all centered and stochastic processes with covariance kernel K' indexed by X' that satisfies conditions (i) and (ii) above. With a slight abuse of notation, we will also use P(K) to denote the set of all distributions over (x(() induced by these stochastic processes. Each element ja E P(K) is now probability distribution over (X).
Let R. C denote the set of real and complex numbers re- spectively. The set N and Z denote the natural numbers and integers. Define No- == N U [0]. For any R2, we use help to denote the 8 norm of x. Given set X' C Rd, we use (2)) to denote the space of squared integrable real- valued functions on X' under some base measure v. For any u E (2), we define ||u| == Jx (=( de(x). The notation V/ is reserved for the base measure on X. whereas u will be used to denote the probability distribution over (X²). For a linear operator F : (X) + (X)), we define |> sup [F ||ellaa = 1]. We use GP
2.3. Problem Setting and Goal
Let F: : (x) (2) denote the operator of interest. One should think of F as the solution operator of the PDE, The goal is to estimate a surrogate Fa using n input/output functions such that
to denote Gaussian Process.
2.2. Distribution Over Function Space
Let X' C Rdd be any compact set, ((()) denote the Borel sigma-algebra, and y denote some finite measure on X (that is, v(2) < oo). Generally, we will take v to be Lebesgue measure on X' but sometimes it may be use- ful to take a weighted measure such as x e e= dr. Denote (2 (())),)) 2) to be the set of all squared inte- grable functions on X. From here on, we will drop the dependence on (()) and v, and just write (x)). Let (2, Σ, P) denote a probability space. We will consider a sequence of real-valued random variables {h+ defined over the probability space (52, Σ, P) that is cen- tered, squared integrable, and has continous covariance ker- nel K= : X' x X' + R. Recall that covariance kernels are symmetric and positive definite. More precisely, for any n, E X, the random variables h2 satisfies
sup E onp
- -
(I)
AEP(K)
is small. In the absence of additional knowledge about the image space of the solution operator F, minimizing this ob- jective is the most natural choice. Accordingly, for fixed μ., the LJ B--chnner norm has been standard error metric in the operator learning literature (see (Kovachki et al., 2023, Section 2.2), (Liu et al., 2024)). The p = 2 case, in particu- lar, is of practical significance, as its empirical counterpart is the widely used mean squared loss. Regarding the family of probability distributions, our proposed family P(K) aims to unify and generalize marginal distributions on input func- tions commonly used in practice (Li et al., 2021; Lu et al., 2021). This family also aligns with the recommendation of Boullé & Townsend (2023). Other families of probability distributions, such as the set of all compactly supported measures on a Hilbert space, have been used in theoretical analyses (e.g., (Liu et al., 2024)). Extending our result to include other distribution families of theoretical or applied interest is left for future work.
X}
/ h hw(w) (w))(P(w)) dP(w)= =0
B[ha]=
Elh2] = [ [ dw) < 0a
Elh, hy] = / (w) dP(w) K(v,z).
Throughout this work, we will assume that the learner knows the covariance kernel K.
Assumption 2.1. The learner knows the kernel K.
Next, we use this process to define probability distribu- tion over (x'). To that end, it will be more convenient to write the process as a function h : X' x Ω + R. By defi- nition, h(x,,) is E-measurable for every x E X. However, this is not enough to argue that is a random element of (x'). Thus, ensure measurability, we will only con- sider stochastic processes h that satisfy the following: (i) The process h is measurable with respect to product sigma algebra B(x) x Σ and (ii) For every w E Ω the sample path h((,w) : X + R. is an element of (2x).. Conditions (i) and (ii) ensure that w H A(:,w) is a measurable function from Ω to (2() (Hsing& & Eubank, 2015, Theorem 7.4.1). In other words, h is a T.2 (X) valued random variable. We
Although not always explicitly stated, this has been a stan- dard assumption in the operator learning literature. For example, Li et al. (2021) and Kovachki et al. (2023) gener- ate their input functions, both during training and testing, from a Gaussian process with the covariance kernel K such that its associated integral operator is al a(-v² + +31) for some constants a, 3, y >> 0. Thus, all the empirical perfor- mances observed in these works are in a setup similar to those described above. Additionally, Boullé & Townsend (2023, Section 1. also suggests generating source terms (input functions) from Gaussian processes with standard covariance kernels such as RBF, Mattern, etc.
3

---

On the Benefits of Active Data Collection in Operator Learning
Additionally, from a teamningtheoortic perspective, assum- ing knowledge of the kernel K is arguably without loss of generality, In active learning, it is common to as- sume access to an unlimited pool of unlabeled samples "1,-.., Um ii 14, where μa E P(K), and focus on mini- mizing label complexity-the number of labeled samples requested (Hanneke, 2013). This aligns with our setting, where labeling (e.g.. solving PDE) is the primary cost. Given such unlabeled samples, one can estimate the covari- ance operator as
function g : oo) + oo) such that (1) is bounded below by g(t), even as - co. There is extensive liter- ature that attempts to quantify e for various oracles (PDE solvers), and we can use these results readily to establish bounds on the irreducible error in our context. For example, for spectral solvers truncated to the first N basis functions where the input and output functions are s-times continu- ously differentiable, we typically have E ~ N-s// Here, d is the dimension of the domain ?
3. Upper Bounds Under Active Data Collection
mm
1
Σ ==1
oo -mm) @ (ei- - -mm)
In Section 2.3, we discussed the problem setting and the goal. Next, we specify how the learner can collect the training data (#1; wi)..... ,0).....,',, wa). In a departure from the standard statistical learning setting, where the training data is obtained iid samples from the distribution under which the learner is evaluated, we investigate active data collection strategies. In active data collection strategies, the learner can pick any source terms ,,,..., Un and use the oracle to obtain w, = (v;). Since the goal is to provide guarantees under samples from the distribution μ E PPKK, ) the learner can use the knowledge of K to pick source terms. For a given oracle with accuracy E, covariance kernel K, and the desired accuracy 1) 0, the goal of the learner is to develop an active data collection strategy for the source terms and an estimation rule to produce F such that the accuracy of 17 can be obtained with the fewest number of oracle calls. Or equivalently, achieve an optimal decay in the upperbound of (1) for n N number of oracle calls. Under this model, we provide an upperbound on (1) when F is a bounded linear operator.
where F. 1
m Ei=1 U4. Since ||||=111 < oo, Theorem 8.1.2 of (Hsing& & Eubank, 2015) guarantees that En,. + Σ almost surely in Hillert-Schhiie norm, where E is the inte- gral operator associated with K. While our work assumes E has a finite trace norm, this is not required to recover its eigenfunctions: convergence in Hilbert-Schmidt norm suf- fices for accurate spectral approximation. Thus, assuming access to the eigenfunctions of K is reasonable in theory, even if it may be computationally demanding in practice.
Once the input functions are generated, the learner has to use numerical solvers to PDE numerically in order to gen- erate the solution function. In this work, we will make the following assumption about learner's access to the PDE solver.
Assumption 2.2. The learner only has black-box access to F through an e-approximate oracle O that satisfies
sup [1O(v) - F(v)lli+ Se. (x)
Theorem 3.1 (Upper Bound), Suppose F is a bounded linear operator. There exists deterministic data collection strategy and a deterministic estimation rule such that the estimate Fr, produced after n. calls to oracle O satisfies
From implementation standpoint, it might seem unnatural to consider Ovv) for function E (22) especially since most PDE solvers usually only take function values over a discrete grid as an input. Nevertheless, the oracle is an abstract object, and the grid can be integrated into its definition. For example, given any function v, the oracle first extracts the values of v on a grid (.1...., and produces output values on the same side, the oracle may then construct an actual function, either through trigonometric interpolation or simply by setting the function values to zero outside the grid points. Thus, we do not specify these implementation details of the oracle and instead characterize it solely by accuracy parameter 8.
sup E. MEP(K) amps
<
E1+1F
Σ A
Here, A1 2 2 are the eigenvalues of the integral op- erator of K defined as (Ixu)()) = =Ix K(.,x) u(x) dv(x)).
different grid. On the output
The first term above is the irreducible error, which depends on the quality of the training data. For the second term, Hsing & Eubank (2015, Theorem 4.6.7) implies that
Σ A1 S>==f(K(rr,)) du()) < suup [K(x,)) x)) u()) < co. 1=1
xc
In general, E primarily reflects the discretization error for knite-difference type methods and truncation error for spec- tral methods, but it may also include measurement errors or errors resulting from the early stopping of some iterative routine. Therefore, & can be broadly viewed as quantifying the quality of the training data. From this perspective, E rep- resents the irreducible error in (1). Specifically, there exists
This is finite because is finite measure on X, and K(x, x) is a continuous function on compact domain, making it bounded. As result, the second term in the upper bound of Theorem 3.1 vanishes as n - 30. In Section 3.3, we apply Theorem 3.1 to derive precise rates for several common covariance kernels.

---

On the Benefits of Active Data Collection in Operator Learning
3.1. Data Collection Strategy and The Estimator
There are two primary challenges in completing this argu- ment in fully rigorous manner. First, must address the fact that the oracle O is only e-approximate for any > 0. Second, the convergence statement for Σj=1 (+) is quite specific, requiring careful attention when applying this result.
Here, we specify the data collection strategy and the estima- tor that achieves the claimed guarantee in Theorem 3.1. Let {y }=) be the sequence of eigenpairs of K defined by solving the Feldholm integral equation
f Kly, x) (p)(())) X3 (V)),
y. E X.
3.3. Examples of Covariance Kernels
Given the Oracle call budget of n, the input functions that the learner selects are (I,, ..,,..., SP as source terms. For each& i E [n]. the learner makes an oracle call and obtains w; = (pp.). Then, we consider the estimator
To make the upperbound in Theorem 3.1 more concrete, let us consider a few specific covariance kernels K. While not all claims are rigorously proven in this subsection, a detailed and formal treatment of the material can be found in Appendix B.
F
3.3.1. FRACTIONAL INVERSE OF SHIFTED LAPLACIAN
Li et al. (2021); Kovachki et al. (2023) generated input functions from GP(O, a(- 31 >) for some constants a, 3, > 0. Here, V is the Laplacian operator defined as
More precisely, this estimation rule yields an operator Fr such that FRv= = Ei= wa (pp,,) u) for any v E (2)).. Appendix A.I provides an overview of the process for de- riving this estimator starting from least-squares estima- tion rule. Furthermore, Appendix C discusses methods for approximating the eigenfunctions ; when the Fredholm integral equation cannot be solved exactly.
02v
V
Ox²
j=1
In this section, we will consider X to be a d-dimensional periodic torus TV and the base measure v is Lebesgue. We identify T by [0, 1] with periodic boundary conditions. Let us define a function ymm T - C as ym (m e i m-w for every m zd. Recall that (m is the eigenfunction of 2 with eigenvalue - -4x² 2 |m
3.2. Sketch of Proof of Theorem 3.1
We now provide high-level, non-rigorous sketch of a proof of Theorem 3.1, and defer full proof to Appendix A.
To bound the risk of the estimator specified above, we first rewrite the risk using Karhunen-LLoéve Theorem. Pick any μ. Since v is defined using a centered and squared- integrable stochastic process with continuous covariance kemee K, the celebrated Karhunen Loève Theorem (Hsing & Eubank, 2015, Theorem 7.3.5) states that
In particular,
82 dx?
V2 2222imm
Σ 1=1
2x 2rima 14 2
Since {smm : m E 2dd forms a complete orthonormal sys- tem in (TTT)), there are no other eigenfunctions of V2, simple algebra shows that Pm are also the cigenfunctions of (- + BI) -7 with eigenvalues being (.3 + 4m mm|3)
v(-) Σ ().
where Ej's are random variables defined as E, J Vx fx oxx) (r)()) dv(x). It turns out that E are uncor- related random variables with mean O and variance 1. That is, Eic,] = 0 and Efg, E,] = 1 =]].
Using this fact in Theorem 3.1 yields the upper bound
a ||
56² <=2 (a8
(a+ +a+
1
27 d
27
This decomposition allows us to rewrite expectation over f in terms of expectation over the randomness of the sequence (E))221. which is more tractable. For simplicity, assume that e = 0. Then, using Karhunen-LLovve expansion, we can show that
When 27/d > 0, the reducible error above goes to 0 when n + cc. Again, as an example, Li al. (2021) uses a 3/2 3 = 49 and 7 2.5 in their experiment for Navier Stokes. In this case, 2y/d 2.5, yielding the convergence rate of n -15 for the reducible error. Note that this rate is faster than the usual passive statistical rate of L/n. However, for any value т, one can take ? = d(r + 1)/2 to get the rate of n T. Thus, every polynomial rate is possible depending on the choice 7f
Exrnye [ [0] s Be (Eim ( 93
which can then be further upper bounded by |[ Σj>n X; using properties of &/'s.
5

---

On the Benefits of Active Data Collection in Operator Learning
3.3.2. RBF KERNEL
query of an input function within an appropriate function space.
Let X R and (x, exp (- 277 ------y for all y,y R and é > O. For now, let is Gaussian measure with mean 0 and variance x² on R. Using the known results on eigenfunctions of RBF kernel in terms of Hermite poly- nomials (Williams & Rasmussen, 2006, Section 4.3.1), we show that there exists 7 E (0, 1) such that the upper bound in Theorem 3.1 is
4. Lower Bounds on Passive Learning
In this section, we establish a lower bound on (1) for any passive data collection strategy. As usual, the ker- nel K is known the learner. Nature selects a distribu- tion μl, E P(K), and the learner receives n i.i.d. samples 11, ,2,...,Uu ~ p... For each i [nl, the learner queries the oracle O to produce w; = (w:). It is important to emphasize that the learner can only make oracle calls for the i.i.d samples 11, V,,..., on. If the learner were allowed to make oracle calls for other input functions, the learner could simply disregard these i.i.d. samples and implement the "active strategy" from Section 3.1. Such restriction on oracle calls still includes most passive learning rules of interest, such as arbitrary empirical risk minimization (HRM), regu- larized least-squares estimators, and parametric operators trained with stochastic gradient descent.
< (1-7) (e [[Fl )*)
That is, the reducible error vanishes exponentially fast as n - co. In Appendix B, we also show that a rate faster than any polynomial rate can be achieved for RBF kernel on R
3.3.3. BROWNIAN MOTION
Let us consider the case where X' = [0, 1]. the base mea- sure V is Lebegsue, and the stochastic process in Section 2.2 is Brownian motion. Recall that the Brownian motion a Gaussian process with covariance kernel K(s,t) = min(s, t) for all s,t [0, 1]. It is well-known (Hsing & Eubank, 2015, Example 4.6.3) that the eigenfunctions of K can be written in terms of sine waves. A simple analysis can then be used to establish an upper bound of
Using these n training points {(E, w;) the learner then constructs an operator F,.. Since the learner only has access to samples from j4+, it is unrealistic to expect a uniform guarantee over the entire family P(K) as established in Theorem 3.1. Therefore, in this section, the learner will be evaluated solely under the distribution μ.. The objective is to minimize the expected loss under μ4,, defined as
e2 2 + 1
2 op x² 2n 1
E Vio
E pmp.
Therefore, the reducible error vanishes at rate n n1
3.4. Comparison to Traditional Active Learning
Moreover, establishing any meaningful lower bound on this risk requires imposing some restriction on the oracle O. To understand why, consider the case where F is a finite-rank operator that only maps to the span of (1/1, 1/2,..., NNn) ) for some orthonormal sequence ii,,.., ,uN in (2))). Now, consider an oracle O such that for any U L (XY)),, it outputs
The active learning framework we adopt in this work is re- ferred to the membership query model, which has long- standing history in the learning theory literature (Angluin, 1988; 2001). However, in traditional learning settings, the membership query model where the learner can request labels for any unlabeled instance-is generally unrealistic. For example. in the context of human data, it may not be feasible to generate label for an individual with an arbi- trary feature vector, as such person may not exist in reality. As result, other active learning frameworks, such as the stream-based sampling model (Atlas et al., 1989) and the pool-based model (Lewis & Gale, 1994; Hanneke, 2013), have gained prominence in the recent literature. These mod- els restrict the learner requesting labels for instances sampled from a specific distribution, making them more practical for many real-world applications. For a compre- hensive review of active learning models, their history, and key results, we refer readers to (Settles, 2009). That said, we believe that the membership query model is the right model for developing surrogates for solution operators of PDEs. This is because a PDE solver can provide solution to any
O = F(t) + X
where X E R and v/4+1 is a unit norm function in (2)) that is orthogonal to all 3 for 1 م N. If |f S E, it is easy to see that
sup 110(v) - F(u) = Ilx @x = Ixl² S e². (=
Thus, O is valid oracle according to Assumption 2.2. How- ever, in principle, it is possible to encode the entire identity of F in real number x. Thus, the learner could determine the identity of F with just a single call to O making any attempt at establishing a lower bound futile.
This problem may still persist even when e = 0. Consider the case where v is the Lebesgue measure, and the oracle
6

---

On the Benefits of Active Data Collection in Operator Learning
of the form
5. Experiments
In this section, we conduct numerical studies comparing our active data collection strategy with passive data col- lection (random sampling) for learning solution operators for the Poisson and Heat Equations. For the actively col- lected data, we implement the linear estimator defined in Section 3.1. On the other hand, for passively collected data, we use a least-squares estimator, where the pseudoin- verse is computed numerically. Recall that, given input- output functions {1; Ws )s) the least-squares estimator has a form -= (E wi @ vs) (-= V @ v.).. For the actively collected data in Section 3.1, the v, " are orthogo- nal, which yielded a simple and natural pseudoinverse (see Appendix A.I). However, for the passively collected data, the vt's may not be orthogonal anymore and the pseudoin- verse does not have a nice closed form. Thus, we use stan- dard numerical techniques to compute the pseudo-inverse XE V/ @ v.). However, in practice, one rarely uses lin- ear estimators for passively collected data. Thus, we also compare our method against the Fourier Neural Operator (Li et al., 2021), the most popular architecture for operator learn- ing. Our code is available at https: https:///åtthb: com/ unique-subedl /ttpa/avveeoreatrreeeeeeeeeeeeee
( - F(u) + x11r= =00]
for some To X. Then, for any v I. (2), we have |[O())
|xx = zoHll = 0 as v((zu)) = 0. This shows that the oracle can still reveal the identity of F in regions of the domain that have zero measure under Therefore, to avoid these pathological edge cases, we will assume that the oracle is perfect.
Definition 4.1 (Perfect Oracle). O is a perfect oracle for F if, for every v E (x), we have
- (t)
Vx X.
In other words, the perfect oracle O produces exactly the same function that F does
more, nothing less. With this assumption, we are in the usual realizable setting often considered in statistical learning theory. That is, the learner has access to n samples ((v,, F(e:)) where U1,-.., Un are drawn iid from some distribution jr.
Theorem 4.2 provides a lower bound on the risk of any estimator under such passive data collection strategy.
Theorem 4.2 (I (Lowerbound). Fix any continuous covari- ance kernel K with eigenvalues A 2 X2 2 .... Then, there exists a solution operator F, accessible to the learner through a perfect oracle O, such that the following holds: for every N, there exists distribution ps P(K)) such that, under any estimation rule within passive data collection strategy, the risk of the resulting estimator Fr is
5.1. Poisson Equation
L't = (0, 1J2. Consider Poisson equation with Dirichlet boundary conditions:
-v-u= u(r) 0 Vr E: boundary(YY).
where V is the Laplace operator. The objective is to learn the solution operator that maps the source function f to the solution u. This solution operator is the inverse of the Lapla- cian, which is a compact linear operator since bounded. For the passive data collection strategy, the input functions are independently sampled as ~ GP(O, 507(-+++ where GP denotes Gaussian Process.
||a((0)
- F(w)
11 L2
m Aj
E Druge
2
1i "
2
for every fixed m E N.
The solution u is computed using the finite-difference method. Both linear estimators and Fourier Neural Op- erators (FNO) are trained on n such independently sam- pled pairs (f, u). For testing, 100 additional source func- tions f ~ GP 502 + +1)++3 (-)) 2) are generated, and their corresponding solutions are also obtained via the finite- difference method. Both active and passive estimators are evaluated on this test set, with the performance measured using the mean-squared relative error:
Specifically, for m 1, we obtain a lower bound of 11F11 A1. This provides a non-vanishing lower bound for any non-trivial operator F and covariance kernel K.
Our lower bound is constructive: we explicitly define a diffi cult distribution for the learner. We construct a distribution over input functions such that, along each eigenfunction direction joj, it places mass with probability 1 - p, and #1/ with equal probability p/2. This yields a sparse distribution with rare but large spikes. A careful argument shows that this construction defines valid distribution in P(K) for any p >> O. When p is small, the learner observes mostly zero inputs during training with probability at least 1/2, yet the expected squared error along each direction is (I/ p 1, leading to a non- vanishing error. The full proof is provided in Appendix D.
predicted
Error
Fleest
true
12
We report the relative error instead of the absolute error to normalize for potential arbitrary scaling due to the norms
7

---

On the Benefits of Active Data Collection in Operator Learning
of the true solution function. The FNO model has four Fourier layers and 2/2 Fourier modes, where N denotes the number of grid points along each spatial dimension. In our experiments, all computations are carried out on 64 x 64 grid, so N = 64. Figures (1) and (2) show the testing error as function of the training sample size. The performance of FNO on active data is not included in this figure due to its poor results. However, Figure (5) in the Appendix includes the error curve for FNO trained with active data, alongside the results for passive data.
the convergence rate of estimator in log scale, using ac- tively collected data across various values of yy-
10a
10-1_
10-2-
10-3
Relative
10-1
y=
10
Linear Estimator (Active)
Linear (Pessive) FNO (Passive)
10-3
1.5
0.8
y=2
10
Pelatiie error
10a
10a
Training sample
Figure 3. Convergence rate of the active linear estimator Pois- equation with actively collected data for different values of
04
5.2. Heat Equation Consider the heat equation
50
100
150
200
Training sample size
Ou V
Øt
Figure 1. Error Plots for various estimators for Poisson Equation. The blue curve shows the performance of our linear estimator actively collected data. The orange and red curves include the linear estimator's ss and FNO's performance passively collected data. Figure 2 shows the same plot in log-scale.
where u [0, + R vanishes on the boundary. The solution operator for this equation is given by exp(rtV2)) and the solution at time t 2 can be expressed as ue exp(rtV²)) ( Fixing 1. our objective is learn the solution operator exp( exp(rV²). This operator is defined as
(TV2)*
exp( V2)
10a
k!
k=
10-1
which is a bounded linear operator. As in the previous case, we sample n initial conditions un ~ GP(O, (-V2 For each initial condition, we use the finite difference method with forward-time discretization to compute the solution V1 at t
Relative tessing error
10-3
This done using 1000 time dis- cretization steps 64 64 grid. For our experiments, set 10-2 As is the step size in the forward Euler method, choosing a larger 7 would result in instability in the numerical PDE solver.
10
3
Linear Estimator (Active)
Linear Estimator (Passive)
FNO (Passive)
10P
10a
10a
All estimators are evaluated on a test set of size 100, drawn from the same distribution as the training data. Figure (4) presents the relative testing errors. Furthermore, the error plot for the Fourier Neural Operator (FNO) trained ac- tively collected data is shown in Figure (6) in the Appendix. Finally, Figure (7) in the Appendix shows the convergence rates of the active linear estimator for different values of
Training sample
Figure 2. Error Plots for Poisson Equation in log-log scale.
While the convergence guarantee of our estimator is formally established only for the covariance operator with > as = 2, we observe that the estimator demonstrates robust convergence even when 7 S in the context of Poisson equation. Figure (3) presents
Our experimental results verify the theoretical advantage of active data collection strategies over passive sampling, as
8

---

On the Benefits of Active Data Collection in Operator Learning
References
10a
Angluin, D. Queries and concept learning. Machine learn- ing. 2:319-34 191-99,22, 1988.
10-1
Relative testing eто1
Angluin, D. Queries revisited. In Algorithmic Learning Theory: 12th International Conference, ALT 2001 Wash- ington, DCC USA, November 25-28, 2001 Proceedings 12, pp. 12-31. Springer, 2001.
10-2
10-4
Atlas, L., Cohn, D., and Ladner, R. Training connectionist networks with queries and selective sampling. Advances in neural information processing systems, 2, 1989.
Linear Estimator (Active)
4
FNO (Passive)
10a
Azizzadenesheli, K. Neural operator learning. International Conference on Machine Learning (Tutorial), 2024.
Training sample size
Figure 4. Error Plots for the Heat equation in log-scale.
Baker, C. T. and Taylor, R. The numerical treatment of integral equations. Journal of Applied Mechanics, 46(4): 969, 1979.
Bhattacharya, K., Hosseini, B., Kovachki, N. B., and Stu- art, A. M. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121-157, 2021.
established in Theorem 3.1. These findings highlight the practical utility of active learning frameworks in improving data efficiency for operator learning tasks.
Boullé, N. and Townsend, A. mathematical guide oper- ator learning. arXiv preprint arXiv:2312.14668 2023.
6. Discussion and Future Work
Boullé, N., Halikias, D., and Townsend, A. Elliptic pde
In this work, we show that arbitrarily fast rates can be achieved with an active data collection strategy when the op- erator of interest a bounded linear operator and the input functions are drawn from centered distributions with contin- uous covariance kernels. natural extension of these results would involve non-linear operators. Specifically, one might ask whether there exists natural class of non-linear oper- ators that permits such fast rates when input functions are drawn from centered distributions with continuous covari- ance kernels. natural starting point might be to consider the RKHS of operators. Additionally, given that functional PCA is the estimation of truncated Karhunen-Looks composition, would be interesting explore whether variant of PCANet-based architecture could achieve fast rates with active data collection.
learning is provably data-efficient. Proceedings of the National Academy of Sciences, 120(39))e22003944120 2023.
de Hoop, M. V., Kovachki, N. B., Nelsen, N. H., and Stu- art, A. M. Convergence rates for learning linear opera- tors from noisy data. SIAM/ASA Journal on Uncertainty Quantification, 11(2):480-511 513, 2023.
Hanneke, S. A statistical theory of active learning. Founda- tions and Trends in Machine Learning, pp. 1-212, 2013,
de-
Hsing, and Eubank, R. Theoretical foundations of func- tional data analysis, with introduction linear opera- tors, volume 997. John Wiley & Sons, 2015.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 149 2023.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning for scientific applications. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Kovachki, N. B., Lanthaler, S., and Mhaskar, H. Data complexity estimates for operator learning, arXiv preprint arXiv:2405. 15992, 2024a.
Acknowledgements
Kovachki, N. B., Lanthaler, S., and Stuart, A. M. Oper- ator learning: Algorithms and analysis. arXiv preprint 1mXiv:240.155755 2024b.
We acknowledge the support of NSF via grant DMS- 2413089

---

On the Benefits of Active Data Collection in Operator Learning
Lanthaler, S. Operator learning with pca-net: upper and lower complexity bounds. Journal of Machine Learning Research, 24(318):1-61 2023.
Lanthaler, S., Mishra, S., and Karniadakis, G. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6())tmac001, 2022.
Lewis, and Gale, W. A sequential algorithmfor training text classifiers. In SIGIR 94:: Proceedings of the Seven- teenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp. 3-12, 1994.
Li Yu, X., Xing, W., Kirby, R., Narayan, A., and Zhe. S. Multi-resolution active learning of fourier neural opera- tors. In International Conference Artificial Intelligence and Statistics, pp. 1440-2448. PMLR, 2024.
Li, Z., Kovachki, N., Azizzadenesheli. K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, Fourier neu- ral operator for parametric partial differential equations. International Conference on Learning Representations, 2021.
Liu, H., Yang, H., Chen, M.. Zhao, T., and Liao, W. Deep nonparametric estimation of operators between infinite dimensional spaces. Journal of Machine Learning Re- search, 25(24):1667, 2024.
Lord, G. J., Powell, C. E., and Shardlow, T. An introduction to computational stochastic PDES, volume 50. Cam- bridge University Press, 2014.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via deeponet based the universal approximation theorem of operators. Nature machine intelligence, (3):228-222) 2021.
Musekamp. D., Kalimuthu, M., Holzmüller, D., Takamoto, M., and Niepert, M. Active learning for neural pde solvers. arXiv preprint arXiv:2408.01. 01556, 2024.
Nelsen, N. H. and Stuart, A. M. The random feature model for input-output maps between banach spaces. SIAM Journal on Scientific Computing, 4365):A3212-A3243, 2021.
Settles, B. Active learning literature survey. 2009.
Subedi, U. and Tewari, A. Error bounds for learning fourier linear operators. arXiv preprint, 2024.
Williams, C. K. and Rasmussen, E. Gaussian processes for machine learning, volume MIT press Cambridge, MA, 2006.
10

---

On the Benefits of Active Data Collection in Operator Learning
A. Proof of Theorem 3.1
A.I. Specifying Data Collection Strategy and The Estimator
We first specify the estimator that achieves the claimed guarantee in Theorem 3.1. Let {Ay, (Ayx491]==) be the sequence of eigenpairs of defined by solving the Feldholm integral equation
Jx K ((,,z) dv(a) = (y))),,
U E X.
Given the Oracle call budget of n, the imput functions that the learner selects are G1 92. Pn as source terms. For each i [r)]. the learner makes an oracle call and obtain
W/ = O(pi).
Consider the estimation rule
arg min
Σ Lips - will2 1122
L is linear
1=1
Solving this optimization problem boils down to solving the linear equation
Σ w. 8 Lo
Σ p. 2 Pi
is clear that this system is ill-posed and has infinitely many solutions. The family of solutions can be written as
n
Σ
sps
2 pi
where i indicates the pseudoinverse. Each particular choice of pseudoinverse yields distinct solution. Since yi 's orthonormal, natural one is
n
Σ 9i
Σ
pi
p 2 pr.
This choice of pseudoinverse yields the estimator
8. = Σ wi @ Pi:
which will be our estimator interest.
A.2. Rewriting Risk using Karhunen-Loove Theorem
Next, we bound the risk of this estimator. Pick any Since e is defined using a centered and squared-integrable stochastic process with continuous covariance kernel K, the celebrated Karhunen-Loève Theorem (Hsing & Eubank, 2015, Theorem 7.3.5) states that
v(-) = Σ
VT ==1
where Ey's are random variables defined as
5, = J / (x) (z) dv(x).
Here, Uz (w) is simply just v(x), but we write the dependence on w explicitly to highlight the fact that v is generated by stochastic process on the probability space (2,, E.P).

---

On the Benefits of Active Data Collection in Operator Learning
turns that Ej's is are uncorrelated random variables with mean and variance 1. In particular, we have
[cy 0 and E 1 j].
The precise convergence statement is
lim sup E 00000 TE
u(x) E
O.
j=1
We refer the reader to standard texts (Hsing & Fubank, 2015, Theorem 7.3.5) or (Lord et al., 2014, Theorem 7.52) for the full proof of Karhunen-Loève Theorem.
Fix m E N such that m > n and define II to be a projection operator onto the first m eigenfunctions of K. That is, for each e with Karhunen- decomposition v(-) = Ej=1 V SJ (r((), we define
(m) = Σ (5E
j=1
Since both Fn, and F are linear operators, we can write
(v()) - F(w))) 110
E tmpp
(I (Im(v)) - -I([Im (v)) + -Fn (n) (v))
=
E
|| (Imm (v)) - -F (v))
2
+ +2 (Tm) (()m(()) |f, - (u- [mm(()) 12
<
E
(
+
E
The inequality follows upon using triangle inequality and expanding the square. For the cross term, Cauchy-Schwar inequality yields
E
Fr, (Im(v)) - F (mm(())) (2)) 12
(fa-F)) Im(())
Fr (m) F F(Im(())
12]
1B [|( [u- mm(())
E
12
L7
Thus, we can write
E pryfe
-
51 + (1)) (I),
where we define
Fr [T (m)) F F(Im (1)) (mm) 1?2
(I E
(Fr, F) II (w)) |2
(II = E
Next, we will bound (I) and (II) separately.
A.3. Bounding (I).
Pick any v ~ μr. Then, we know that there exists {Eybee such that Ee which subsequently implies that
VT So, (1) =E VTy
(1)) Fr
E =
= Σ マソ
==1
12

---

On the Benefits of Active Data Collection in Operator Learning
where the final equality uses the fact that n m and Fr (1,) 0 for all j > n. Defining 6, == ==(())) F(Ps)) we obtain
wy F(P)) 8,. This allows us to write
(Im((v)) = Σ
+60)
=F Σ /e
Σ g.d.
F (1)) - F
Σ vayer9s
+ EVVgs
j=n+1
So, can rewrite (I)
1 1.2
E vxp
F,, (v)) - F F(Imm(1))
E
Ve -F
Σ
ViyEsP3 Ps
1=++1
VAJSSPs) )
)
2
= E
M &
-
-2E
Σ
V
+
F
Σ
The cross-term vanishes upon swapping sum and integral as Él's 's are zero mean and uncorrelated. For the first term, note that
E
Ax & 8,
E
Ae E di
L
\ EKS ||امl +2 Σ
V VA,Y BLEE 6,),) 12
Igiejsn
A &100lli:2 +
< A
The final inequality uses the fact that O is e e-approximate for F. For the third term, similar arguments show that
m Σ
S
F 2
Ey E 4j
j=n+1
L2
Ø=++1
m
| 2
Σ
A 19331
j=++1
m
| op
Az-
j=++1
Thus, we have established that
m
())
S c2 Σ
\ +
|| 2
Σ j=n+1
lp
1=1
13

---

On the Benefits of Active Data Collection in Operator Learning
A.4. Bounding (II)
For any 0 ~ μ., have
E
( [
F))
1m (v))
Fr
=11 op
E
1I
(2)1
12
Let E21 /ggE.ps Then,
E
|Iu (m)))
= E
U Σ
3=1
m u(x) Σ
E
Va (2((0)
de(z)
j=1
A Es ( /7 de(()
//2 E
v(x) Σ
m
x
j=1
S x()) sup E xE
v(z)
The third equality uses joint measurability, finiteness of v,, and Tonelli's 's theorem to exchange the order or integration. Therefore, we have established that
(II)
S
Fr - Fl| up
v(X).
sup E sEt
v(x)
V (A - Σ
j=1
A.5. Combining (I) and (II) For each m n, we have established
[u(()) - F(u) [1]
E vnuu
< (1) + (1()) (II + (II)
where
m
(1 < 2
Aa | 2
Σ
up
j=
u(x)) Σ m
(II) < 8r - F
v(x)
sup E BE
(A
It now remains to choose such that the upperbound is minimized. To that end, will take + co. Since Wi,.... Wn co. As F is also a bounded linear operator, for any n E N, we must have
(2)),, we must have [
lopp
op
|.x
F 00.
Importantly, the norm of Fr - F may grow with n, but is independent m and does not grow as m - co. Moreover, as // finite measure, we must have v(2' < co. Therefore, Lorv Theorem (Hsing Eubank, 2015, Theorem
14

---

On the Benefits of Active Data Collection in Operator Learning
7.3.5) (also see Equation (2)) implies that
27 Va Vis
m u(x) Σ
(I)) Σ
F F
v(x))
sup E #Ed
m 0.
op
1=1
On the other hand,
1)
X + | lop
1=1
j=n+1
Overall, we have shown that
E emep
[|
F(w))
S Σ
A + 1
Σ
1/2
i=1
j=n+1
This completes our proof of Theorem 3.1. B. Examples of Covariance Kernels
In this section, we build upon and present a more rigorous analysis of the material discussed in Section 3.3 of the main text.
B.1. Fractional Inverse of Shifted Laplacian
Li et al. (2021) and Kovachki et al. (2023) generated input functions from GP(O, all (x--2 + (3)=>) for some constants a, 3, 7 > 0. Here, v is the Laplacian operator defined as
020
V
Σ j=1
Ox
In this section, we will consider X' to be d-dimensional periodic torus T and the base measure is Lebesgue. We identify T by [0, 1]d with periodic boundary conditions.
Let us define a function pm Td + C as (m((z) = 2rima for every m E Zd, Recall that pm is the eigenfunction of 72 with eigenvalue -4x mm|.. In particular,
2 2n
Σong
Σ 0
2x
Σ (2n Brrmmy) )2 2 - 2m e 2rima
Since {ypm m E Zd forms a complete orthonormal system in (T there are no other eigenfunctions of V A simple
algebra shows that 4m 's are also he eigenfunctions of shifted Laplacian -22 3I with eigenvalues being (3 + 4x | () Finally, the spectral mapping theorem implies that {(Ams Pm) m E zd) is the sequence of eigenpairs of a V2 + +1) where the eigenvalues
Am = (3 4n ml
Next, we need to show that these eigenvalues summable Theorem 3.1. Note that
Σ
Am
Σ mEZd
4x 508
Σ
|m|_2
(2T)) 22
mEZd
mEZd (()
is easy to see that
Σ
|m|022 < 2y (2j + 1)a < 3d
2- < 80
(E)
as long as 2y > d. The first inequality holds because I'mm E Z |m| = 5 < 2(2 + 1)d-1, This is true
because at least one of the entries has to be +j and other 1 entries could be anything in [0, #1. =j. So we have
Emeza |aml < oc, implying that the operator af 2 + (I) -> is in trace class as long as 27 > d.
15

---

On the Benefits of Active Data Collection in Operator Learning
Finally, it is easy that the operator a(-V² + 31) - is integral operator associated with the kernel
((,,))= = Σ
Am Pm (y) pm)0)mmm))))) (z).
mE24
Upon writing the Fourier series of (T)). it is obvious that (a(-V² 31) 31) (v) = fx ()),))((z) dx for all E X. As Emez |m < co, the convergence is absolute and uniform. Since K' is uniform limit of the sum of continuous functions, K must also be continuous. Moreover, as Am = A_ the kernel K must be real-valued. In particular, we have
k(y.z)=
Σ mEZ!
Am (pm(y) 9m (x)
Am (Pm(() (pm(y))Pmmm) (mm(x)) + -- (v) 8-m (x)
Σ mEZd
Σ mE2
Am cos (2rm (y -x))
Σ
Am (cos(2mm 3) cos(2 . 2) + sin(2mm y) sin (2xm +z))
mEZ!
This is generalization of the cosine covariance kernel often considered in computational PDE literature (see (Lord et al., 2014, Example 5.20)). Since cos(f) = cos(-0), . it is obvious that K is symmetric. As K is a continuous and real-valued covariance kernel defined on a bounded domain T we can use Theorem 3.1 for such K. In principle, we could use the Fourier modes im 's as source terms to define the estimator discussed in Section 3.1. However, the proof of Theorem 3.1 assumes that the eigenfunctions of the kernel real-valued. So, will first show that K solely using sine and cosine functions without having to use complex exponentials. This allows us these sine and cosine functions define the estimator discussed in Section 3.1, and invoke results of Theorem 3.1.
write the eigenfunctions of
B.I.I. CASE
When = 1, it is easy to see that {1} U (v2 (2Ttjzzz)) 2 (2mjzz :j E N) are the eigenfunctions of K. Writing the expansion of K and using Fubini's switch the sum and the integral, we get
[ K [(y.a))2cos((2m))) dx Ay XyV2 cos (2Tj)))))) 2 + X_yV2 cos( cos(-22Tjv) = = -)yx2co((2Tjy)
2
Note that the first equality holds because cos(2Tjæ) is orthogonal all other cosine and sine functions except for cos (2 and cos(-22rj) The final equality holds because X; X-j and cos(f) cos( (Q). A similar calculation shows that
[ K K(y,z) 2(y,2))/ssn((2mj)) dx Ag Vyv2 sin(2 + x V2 sin(-2rrjv) 2 = -
Finally, we have f x(),z) 1r 201. Thus, Ay for j E are the eigenvalues for sine/cosine functions and Ào for 1. j E N} forms a complete orthonormal system of (I, R), there cannot be
Since {1} U 2 (2 v2 any more eigenfunctions K. Next, we will plug in the values of X, 's in Theorem 3.1 to get the precise rates.
Pick an odd n E and suppose the n input terms used to construct the estimator Section 3.1 are (1)
16

---

On the Benefits of Active Data Collection in Operator Learning
(V2cos (inn(2)) :j S (n 1)/2). Then, the upperbound is
(n-1)/2
DC
x² 10 +
Σ 22j
|
j=(++
(n 1)/2
S 6
+ 22
Σ
a
(33 + 1n
+
F
a
(B + 10+4m2))) 4x
==l
j=(++1//2
(n-1)/2
<Ea08
L aaa (1
2
Σ
2a
|
j2y
(4m²)²
j2y
j=l
j=(n+1)
1)/2"
$600518 L. Eaa
(4T2)7 + 620 a
2
(n-1)/2
t 27 di +
2a
|| 2
600
2 dt
(4T2))
(4
(n-1)/2
260 7 Eaa (1
2
+
2
Eaa
2a
|| 2
1
2
V7 V7>
(4m²)) 27
(472)7
2
(n
1)27-13
2
Since 2. 2-122-1 s (4 and 22 < (4 the overall error is at most
a || op
1 2
2² a a
27 +
2y - (n 1) for all
Since y > 1/2, the reducible error goes to as n + co. As an example, (Li al., 2021) uses a 625, 3 = 25 and y = in their experiment for Id-Burger's 's equation. this case, we get the convergence rate of n- for the reducible error. Note that this rate of cubic order is faster than the usual passive statistical rate of 1/n. In fact, for any value т, one can take (T 1 to get the rate of nT Thus, every polynomial rate is possible depending on the choice of 7.
1.1.2. CASE [>] 1
Recall that {1} U { {V2 cos() sisi(2zja)) :j E N} are the eigenvalues of K for = 1 with eigenvalues Xy :=
(B 4m 22) -? Define set of functions
& 8
(2sin(2mja)) :j N).
1=1
For each element E &, there exists tuple j = (ji,..., ja) E Ng such that
elx) UR (21) ... Dh (xA-1) (xa).
where vi,, (z.) E {v2 cos((2jjjj)) 22 sin (, } for jr > 0 and V2 for j. = Let us denote the collection of all such functions by Eg. Then, we have & = SeN E,. We prove the following result on the eigenpairs of K.
Proposition B.1. For each j E No. define X3- = a (3 + 1i Then,
U U c) jENg eEE,
the set of eigenpairs of K on T
We defer the full proof of Proposition B.1 to the end of this subsection. First, we use Proposition B.1 and Theorem 3.1 get the precise rate for kernel K. Pick such that the source terms used to construct the estimator defined in Section 3.1 are
&
SENS ld
17

---

On the Benefits of Active Data Collection in Operator Learning
Note that [8o 1 and |& S 2d for all Ljloe= > O. Thus, there are S (r 1)d2d source terms. Then, the upperbound is
S = 00 Σ
2 X + |F
2d X3
bil0>r
= =Ea + 222d
a (B + 4T (1 +2 2d F
Σ a (3 + 4T 113)
Sr
= =6008- 7 + 22d
Σ Sr
a (8+ 4m 1 2d |
Σ
(3 4 [b(
56"0g- + 2d
a (B + k²) (k+ 1) + 2d
F pp
(B+ -2 (k + 1)
a
k=
k=
a2d
a 2d
26'08 + 2dd (4 k=1
hd 1d-1-2 27 + 2dd ||
h 1-
24
k=
a2dd
+ 6² (1 // 10-1-27 dl
a22d
22d
S6:087 + 6 (4m2)y
||
pd-1-27 dl
(4 Jr
S sea8- + è a22d
a22d
a 22d
(4T7)T + t (4m?)² 7 27 - d
|| 2
for all 27 > d.
(4 27 d 2-1-d
Recall that n < (2r + 2) So, we have n2/d/2 1 S
For n 1/d
2 4, we have 2 ///4/4. Thus,
1
2
2
n2
2y 2 22dg22-d Moreover, as 2y > d. we also have (4m2)² 2 2nd Therefore, our upper bound is Note that (4m²y= (2 most
a | op
E² 03 a
+ a
2y - d
2y d
27
Since 2y/d >>>00000000 0, the reducible error above goes to 0 when 73/2. = 49 and y 2.5 in their experiment for 2d-Navier Stokes. In this case, 27/d = 2.5, yielding the convergence rate of n 115 for the reducible error. Note that this rate is faster than the usual passive statistical rate of 1/n. However, as usual, for any value т, one can take 7 = d(r + 1 to get the rate of n Thus, every polynomial rate is possible depending on the choice of 7.
Again, an example, (Li et al., 2021) uses
We now end this section by providing the proof of Theorem B.I.
Proof of Proposition B.I. Since Useng UsEe, {e} forms an orthonormal basis of T.2(TT, IR), there cannot be anymore eigenfunctions of K. Thus, it suffices to show that (Az, e) is an eigenpair for any Ej and j E Ng. To prove this, we will establish that
Σ 11mm/ = Vi E [d] } cos (2mm (y - (x) dr = e,()).
(3)
mEZ4
where ej is an arbitrary element of Eg. Recall that
L.v cos (2rm (y x))e,(x)) do Of 3i such that |mm, A jr.
This is true because if Zi such that |ma] * jis then we can write cos (2mm (y - x)) = cos (2T Eess me(3le (r)) cos(2mmm:( =x:)) sin (2m Eesy mr - =z)) sin(2 - =))). Moreover, ez(x) = vSn (x1) ... Bis (Ia-1). Was (ma). where vhj, 's are either sine, cosine, or a constant function. Our claim follows upon noting that V1, (xt) is orthogonal to both sin(2Tm,,y: Ix)) and cos(2 (os(2mm;(y: - x.)).
Thus, Equation (3) together with the fact that Am = Ay for all m E [k E zd
= jr the eigenpair of K. As j E Nó and ej E E, are arbitrary, this completes our proof.
|k//=/
Vi E [d] implies that (A,, is
18

---

On the Benefits of Active Data Collection in Operator Learning
Now, it remains to prove Equation (3). We will proceed by induction d. For the base case, take d 1. If 0, e, and our claim follows trivially. Suppose O. Since cos(O)
we have
1[ml j} cos (2rm(y =)) 2 cos z)) MEZ
Σ
coss(2tj)) (os(ja)))) + 2 sin(2rry) (ir))...)
If (x)) 2 cos(2 then
J (2 cos (2s(jyy))) cos(2 + sin(2rjy) sin(2rrz)) V2 cos(2 de v2cos(2Tjy).
ex) sin similar calculation shows that
J (2 (2s cos(2xjr)) sin(2 sin((zzz)) (2sin(27ja) dr (2sin(2nju)
This completes our proof of the base case.
Suppose (3) is true for d - 1. We will now prove it for d. Note that
cos (2tm
x)) cos
2π (1 - zii
d-1
d-1
= cos
2x
ma(ii - x;) cos (2rmmawya - xz)) - -sin
2T
Σ
m; (y: - v3))
sin (2mma(ya sta))
First, observe that when summed over all E zd such that |m;l = ju for all [d]. the sine term vanishes. That is,
d
Σ
11mml = ja Vi E [d]
sin
2m
mi - zi)
sin (2 =xa))
m224
1=1
d-1
milyi a:)) )
Σ 1f/ml = Vi
E
[d
1}}
sin
2m
Σ
1{/mal ja} sin (2mmalya xr)) m.EZ
Σ
m
0.
The final step follows here because the term in the second parenthesis above is always There are two cases to consider. ja 0. the summand only has term and our claim holds as sin(O) 0. On the other hand, if ja O, then are have (0) + sin(-O) O.
Therefore, we obtain
Σ 1(1mg = js Vi E [d] cos (2mm (y x²)
mEZ4
d-1
m, mi)) ))
Σ 1fmm = de e [d- 10-1] 1}} cos
2T
Σ maEz
1flmal = jal cos (2mma( xa))
A similar factorization can be done for ej to write
e,(z) = v/ (x1).. (Z)))
where 4b, are either sine, cosine, or a constant function.
However, bi is some eju defined on T. Thus, using the base case, we have
J Σ 1|mal = ja} cos (2rma( xa)) Wia (xx) dxy (us).
m
19

---

On the Benefits of Active Data Collection in Operator Learning Similarly, using the induction hypothesis, have
d-1
d-1
1(mm]== Js wi d 1]} cos 2T
milvi mi))
II
Ue (x:) d(ai,-..,
mEza-1
==1
II y (Ui).
Combining everything, we obtain
d
d
f
Σ 14mm = ja vi E [d] cos (2xm (y x)) II 3. (2:) dx = II 2 (v:)
mEZ4
The final step requires using the factorization of cosine and writing integral over T as product of integral over Td-1 and This completes our induction step. and thus the proof.
B.2. RBF Kernel on R.
Let K be the RBF kernel on R. That is, K Kxx 2) = exp (- उ7 x - y/²) for all x,y E R. For now, v is a Gaussian measure with mean 0 and variance o² on R. Then, known (Williams Rasmussen, 2006, Section 4.3.1) that
K K(x,, v) Ex= Ag (3)()) (y)) where
2a
6 )
A =
aa+b+c a aa+b+
(2(x) = exp(-(c- -a)=)) H,(V 20z).
Here, a (402)-1,b b (269)-1.c= va² + 2ab, and H-) is the Hermite polynomial of order j defined
di
=+(x==(11)) exp(x²) dx exp( exp(-x²) x²))
Note that this is the eigenpairs of K x)) over the entire R, whereas we need eigenpairs over some compact domain X' C R. The eigenpairs of K (y, generally not available in closed form for arbitrary X. However, the variance of the Gaussian measure o² can be tuned appropriately to localize the domain E to appropriate X' of interest. For example, let N' [-1,1]. Then,
[" K ((), dx())
-f100)//aa K(y,z) (x)))
(1,,z) (ar)-/ox 121111
Since f K(y, x) (x) de(x)) {y5);(1), we have
L" K ((y,)) - S f/1 1 =1,, du(x)
x2 du(xr) V V. m (x)())) du(x))
/
1 - y|²
|=|21 exp
du(x),
f2
|k-y// I 1
where the second term is upper bounded by 1 as 90 integrates over the whole domain I. Note that exp
and 1| < 3.9 x 10 12 when o = 0.1. So, a can be appropriately tuned such that (Aj, {));))) is a good approximation the eigenpair of K for our domain X' of interest. Next, we use these eigenvalues to study how the upper bound Theorem 3.1 decays as n oc.
20

---

On the Benefits of Active Data Collection in Operator Learning
Let == ba(a + + c). It is clear that E (0, 1). Since c = va² + 2ab 2 we also have
aabtc S 1. Thus, we obtain
2a
A I 7². Plugging this estimate in the upperbound of Theorem 3.1, we obtain
72
7-
A +
N
S
2
70
cp
1
c²
|| op
7
1 7
1
882 +
|| 7')
(1 - 7)
Therefore, the reducible error vanishes exponentially fast as n + 00.
B.3. RBF Kernel on Rdd Let K(y,a) = exp(-las - --3/(222))) where x, E Rdd Then, is clear that
K (V,
II exp(<12; 22(2 II K, 2;). 1=1
1=1
If (Aij, (ii)sen the eigenpairs of K; under the weighted measure standard Gaussian measure R, then
d
d
[I
II
II pi.)
|ui..e...... 1a) E No
the eigenpairs of K when v is multivariate Gaussian with mean 0 and covariance I This follows immediately upon noting that
d = II / K, x.) Pis (2x) du(x:) II 2m
J K(y x)) II
Xug, Piji (zx))
2-1
Finally, these are the only eigenpairs because the product functions Il Pijy for all possible ii. No form complete orthonormal system of L (Rd) under the base measure v.
Pick m such that m > d, and suppose the n source terms in Theorem 3.1 are {Pig : 1 E [d] and I S m - 1}. That is, we have n = md source terms. So, the upperbound is
Auj + 1|
Σ
II
=0
i==
(1...)))EN
1=1
max(ft...fa))m 2m
The first summation is
m-
Σ
Xija
d m-1 II
Auy
m-1
2 <
/ 1 1- d
(
<
1 (1 7)
=
i==
=1
i=
On the other hand,
Σ (ENN) max(jt....e422m
II 1=1
Aun
Σ
<
pd 7"
<
1 A a dr.
000
(1n...)))EN max()i....fa
m
m-1
The second inequality follows because the number of tuple (fi...., .......... ja) that sum to r is < pdd is easy to see that the integral converges faster than 1/n' for every 2 1. To see this, pick t 2 1. Then, there exists >> 0 such that 7" cr dt 1-d Note that e may depend on 7, d, and t, but it does not depend on r. Thus, we obtain
d n dr Sc
m-1 r d-- dr = (m- 1jdl
m-1
21

---

On the Benefits of Active Data Collection in Operator Learning
Since m n/// this rate is d/ /n for some c'. That is, our overall upper bound is
d' (1 (1- || n²'
for some d' for every 2 1. Therefore, the reducible error vanishes at rate faster than every polynomial function of 1/n.
B.4. Brownian Motion
Let us consider the case where X' = [0, 1]. the base measure v is Lebegsue, and the stochastic process in Section 2.2 is Brownian motion. Recall that the Brownian motion a Gaussian process with covariance kernel
ts = min(s, t)
s,t E [0,
It is well-known (Hsing & Eubank, 2015, Example 4.6.3) that the eigenpairs of K is given by
(( (5-2)
1
(j - 2 and
(t(t) = 22sin
n VjEN.
2
Plugging this in the upperbound of Theorem 3.1 yields the bound
1 2
(j
1))²
-= (j 2) ))²=>
1=1
π
+
+| up
Σ
1
2 72
jmn+l (j - 1)²² " n2
g2
1
1
|| lp
(t- 1/2)3 1/23 dt
72
2²
1
2
|| op
n² 2n 1
Therefore, the reducible error vanishes at rate
n2
C. Numerical Approximation of Eigenfunctions
In Section 3.3, we provided analytic expressions for the eigenfunctions of certain covariance kernels. However, for some kernels of interest, closed-form expressions for the eigenfunctions are generally not available. such cases, numerical approximation is necessary. Here, we will briefly mention some key concepts behind the numerical approximation of eigenfunctions of kernels. The material presented here is based on (Williams & Rasmussen, 2006, Section 4.3.2), so we refer the reader to that text for more detailed discussion and relevant references.
Let (x) p(z) dæ for some density function p. For example, if is Lebesgue measure on [-1, |-1, 1]. then p(z) = 1/4. Then, the solution of Feldolm integral
/
K K(y, (()))))(())dv())) = Asp(()
approximated using the equation
K(y, 2.) (x(;)) (v))
Here, 21, Z2, , NN
NN are iid samples from p. Taking y = .,,..., IN, we obtain a matrix eigenvalue equation
Ku; = "i Uj
where K is a N x N matrix such that [K] = K( 2.)) The sequence (7 u5)521 is the eigenpair of K. Then, the estimator
for eigenfunctions yo and eigenvalues Xy 's are
(x)()) ~ VN [uyle
7j A
N
22

---

On the Benefits of Active Data Collection in Operator Learning
The V normalization for eigenfunction is ensure that the squared integral of y, on the observed samples is 1, That is,
/ 1)(0)
1 E (y()))
1 N
V Muyle VN [uy]: = u, u
As for the eigenvalues, the proposed estimator is consistent. That is, ///N + >3 when N oo (Baker & Taylor, 1979, Theorem 3.4).
The estimator for eigenfunction only allows evaluation on points ,,,..., N used to solve the matrix eigenvalue equation. To evaluate the eigenfunction on arbitrary input, one can use a generalized Nyström-type estimator, defined as
VN
N
p0(4)
K(y, xi) [uy]:-
D. Proof of Lower Bound
Proof. Let (py)se be the eigenfunctions of K. That is,
Jx (()))) ((()) -espe()) V.
We now construct a hard distribution for the learner. Fix some E (0, 1) and let E E denote the sequence of pairwise independent random variables such that
T/P
with probability
E
0
with probability P
1/p
with probability h
Given such sequence, define function such that
v(-)
Σ
Note that
|el|
Σ
<0
3=1
as SuPJEN 15,12 < 1/p and Σラ= XÚ < oc. Thus, is random element in (2)) Let μ denote the probability measure over (x') induced by the random sequence HeJee is easy see that [[](x) for each X. Moreover, for every v.yE E X, have
Elo(r) = Σ (T
Σ
Vy (35558(1n]
M, (z) +2 E
Van, E Ei P.
i<g
M B| (z)
(x)
x)),
23

---

On the Benefits of Active Data Collection in Operator Learning
where the final equality holds due to Mercer's theorem and the convergence is uniform over x, E X. Therefore, we have shown that μ P.)). Let 121 be sequence of iid random variables such that Ø5 Uniform([-1,) Fix > 0 and for each such a E {-1, 11 define
20 F. =
j P1 4g.
j=1
For each m E N. we will show that
c² m
E umojs
Fr (1)
Xj.
2
j=1
Since this holds in expectation, using the probabilistic method, there must be a o² such that
E
E umups
-o
Fo-
[
Aj.
2
Noting that ||F llop = completes proof. The rest of the proof will establish this incquality.
Since [4j}eeN the orthonormal bases of (2)),, Parseval's identity implies that
FF()))
Σ (Fa(o)) j=1
Σ (1)()).)))
Recall that F+9r) cajej,, where F is the adjoint of F. Thus, for any v ~ μs, have
(v, (v)
which subsequently implies
| - Fo())) = Σ//
(3)(0),s)) VT
j=1
Using this fact, we can write
[ Fo(())
1
2
E
E
E
12
(F)(()))s)) co, vxyes Es
E
E
E
-
(F) V
E #lin Jat
E
E
In the final step, we changed the order of integration. Note that drawing n samples of U1,..., Un and drawing a can be done in any order, as they are interchangeable. Finally, the draw of ~ I occurs during the test phase, independent of the previously drawn samples U1.. and
Next, let En,m denote the event such that
(b)) =0
V1 Siiin and 1 S j S m.
Then, we will lowerbound
< (Fn(l))
co, ()) V
E E propee
co,
24

---

On the Benefits of Active Data Collection in Operator Learning
conditioned on the event En First, note that
E E
Σ
E Wiwes
Σ
E
(F.()))))) cos -co v
E
B
(Fa(t).
- V 1
where the final step uses Jensen's inequality. Next, the fact that when the event Fn,m occurs, the learner has no information about 01...., m. This is because the input data shows no variation along the directions spanned by ( PP Given that O is the perfect oracle for Fa, any information provided by the oracle O must be independent of how Fa operates on the subspace spanned by .i....., imm Specifically, for every S. i < nu and 1 S j S m, the output of the oracle O(vs) must be independent of o.y. If this condition holds, then the estimator F, must also be independent of o....., Tm Thus, conditioned on the event Fn,m, for any S S m, we have
| (3,())))))) - ca, Va Es =
E
| (=(()):)))
oYoo,}
(Fn(U))P)) (F.(w))))) -c (F,(0))))) ( +c
= E 2
22 /2y1
= E
The first equality uses the fact that conditioned on a v'fa }. the function Fv(v) is independent of o,. Thus, conditioned on the event E,,m. have shown that
E
Σ j=1
2
2
25
= c
B|
Σ Aj.
j=1
==1
-=1
Therefore, overall lowerbound is
2 Σ
m
m
E
E Omopee
(F()) -c, co,
2
X
P [n.mm] = 2 (1-
p) mm
The final step uses the fact that IP (rEEn [m,m] = (1 -pp) mm It now remains to pick p to obtain the claimed lowerbound, Let us pick
Imn Then, we have (1 - mmn 2 1/2 as long as n 2 1, yielding the lowerbound of
2 m
Az.
2
Since m E N is arbitrary, our lowerbound holds for every fixed m. Noting that [ for every a completes our proof.
E. Experiments
This section presents additional experimental results using the same setup as described in Section 5. The results show that the Fourier Neural Operator (FNO) performs poorly with actively collected data. This is likely because the training data are not i.i.d. samples from the test distribution, requiring FNO to generalize out of distribution when trained on actively collected data.
25

---

On the Benefits of Active Data Collection in Operator Learning
E. Poisson Equation
Linear Estimator (Active)
101
Estimator (Passive)
FNO (Active)
Relettve testing error
10"1
10
10-4
Estimator (Active)
2-
Linear Estimator (Passive)
(Active)
FND (Passive)
10-6
100
10a
102
Training sample
Training sample size
Figure 5. Error Plots for various estimators for Poisson Equation. The plot on the right shows the same plot in log scale.
E.2. Heat Equation
10a
Relative testing error
10-2
10-3
10
10-5
Linear Estimator (Active)
Linear Estimator (Passive)
FNO (Active)
10
FNO (Passive)
100
10a Training sample size
102
Figure 6. Error Plots for various estimators for Heat Equations in log-log scale.
26

---

On the Benefits of Active Data Collection in Operator Learning
10a
0.5
y= 1
10-1
1.5
y=2 2
10-2
Relative testing error
10
10-5
10
10-7
10a
Training sample size
Figure 7. Convergence of the active linear estimator for Heat equation with actively collected data for different values of
27


Reviews:
[{"review_id": "LvoE3LOM0a", "reviewer": "Reviewer_XHLU", "summary": "This paper is in the general area of using AI for PDE. Its goal is to minimize the input-output pairs needed to train such an AI model. The paper proves a new bound on the sample complexity. The results show that the proposed method have arbitrarily fast error convergence rates with sufficiently rapid eigenvalue decay of the covariance kernels.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and for recognizing that the overall direction of our work is interesting and potentially impactful. Below, we address the reviewer’s questions and concerns. \n\n\n* We agree with the reviewer that, in its current form, the scope of our work does not fully capture the complexity of real-world PDE problems. Regarding the comment on the small number of samples needed to recover the operator, we note that this efficiency is possibly due to the *linearity* of the underlying operator and the use of actively chosen inputs. While the setting is idealized, we emphasize that this is the first work to provide rigorous theoretical evidence establishing the benefit of active data collection in operator learning. Extending these ideas to *nonlinear* operators is an important future direction to capture real-world settings, and we hope our work lays the foundation for further works in this area.\n\n\n\n\n\n* Since our theoretical guarantees apply only to linear operators, we focused on standard linear PDEs and did not apply our estimator to nonlinear cases.\n\n\n\n\n*  **“What are the set of functions the proposed method uses to approximate the original function?”** During implementation, our method does not require searching over a pre-specified function class via optimization. That said, as discussed in Appendix A.1, the resulting estimator can be viewed as a solution to a least-squares problem over the space of linear operators, with a specific choice of pseudoinverse.\n    \n\n\n* **\"Does the grid size matter in the evaluation?”**  We observed that our active estimator consistently outperforms the passive baseline across different grid sizes. The choice of $64 \\times 64$ was made primarily for computational efficiency, as we trained 16 separate FNO models for different sample sizes to generate the convergence plots. \n\n\n* **“Is there any baseline for comparison in the active learning field that can be used directly for operator learning?”**  Currently, there is no widely accepted active learning baseline for operator learning. Unlike the passive setting, where empirical risk minimization on i.i.d. samples is standard, active learning strategies are typically problem-specific and do not scale well to the infinite-dimensional setting.  For example, uncertainty sampling and Bayesian algorithms will both require careful extensions to infinite dimensions.", "claims_and_evidence": "The claims are backed up with theoretical analysis and experiments.", "methods_and_evaluation_criteria": "I do not have theoretical backgrounds to evaluate the theoretical aspects of the paper, so my comments will be focused on the experimental setup and results. \n\nOne issue is that the paper is that the functions they have studied are pretty simple and does not reflect the complexity in real science applications. The authors only evaluate two equations. I also don't know whether the problem size is enough. I would imagine for large problem size (e.g., complex functions), it would take much more samples intelligently in order to recover the function. I suggest the authors think more thoroughly about how to empirically evaluate the methods. I think the problem is interesting only at a large scale. If an algorithm can converge using 30 samples only, a human probably can manually write down the function directly. Does the grid size matter in the evaluation?\n\nAnother question is about the experiment setup, what are the set of functions the proposed method use to approximate the original function is not clear.", "theoretical_claims": "I didn't check the correctness of the theoretical proofs.", "experimental_designs_or_analyses": "The evaluation results look reasonable.", "relation_to_broader_scientific_literature": "I think the overall direction is very interesting and can have significant implications.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Do you have evaluation results for more complex functions?\n\nDoes the grid size matter in the evaluation?\n\nIs there any baseline for comparison in the active learning field that can be used directly for operator learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXe2gIleiC", "reviewer": "Reviewer_UK7y", "summary": "This paper proposes an active learning method to learn bounded linear operators from data. This method selects input functions based on the eigenfunctions of the covariance kernel, leading to faster convergence rates. The paper establishes minimax optimal error bounds, showing that active learning can outperform passive learning, especially when the kernel's eigenvalues decay rapidly. Numerical experiments on PDEs like Poisson and Heat equations support the theoretical findings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging and positive assessment, and for recognizing that our work offers significant insights by establishing minimax-optimal error bounds and clarifying when active learning is beneficial. We address the reviewer’s questions below.\n\n\n*  **``Q1: Could FNO benefit from using the actively selected training data chosen by the proposed linear estimator?\"**   Interestingly, FNO performs poorly when trained on actively chosen data for the linear estimator. We include the error curve of FNO trained on these active samples in Appendix E. \n    \n*  **``Q2: The number of training samples in the experiments appears to be chosen empirically. Can they be potentially guided by Theorem 3.1 or any theoretical criteria?\"** Yes, in principle, our bound can guide the choice of sample size. Suppose we want to select $ n $ such that the reducible error term  \n$$  || \\mathcal{F} ||\\_{\\mathrm{op}}  \\sum_{j > n} \\lambda_j $$\nis at most some small  $\\delta > 0 $. Assume the eigenvalues decay polynomially, i.e.,  $ \\lambda_j \\lesssim j^{-p} $ for some $ p > 1 $. Note that $p>1$ is required for the kernel to be Mercer. Then we have\n$$\n|| \\mathcal{F} ||\\_{\\text{op}} \\sum_{j > n} \\lambda_j \\lesssim || \\mathcal{F}||\\_{\\text{op}} \\sum_{j > n} j^{-p} \\lesssim ||\\mathcal{F}||\\_{\\text{op}} n^{-p+1} \\leq \\delta,\n$$\n as long as\n$$\nn \\gtrsim \\left( \\frac{||\\mathcal{F}||\\_{\\text{op}}}{\\delta} \\right)^{\\frac{1}{p-1}}.\n$$\nFor example, when $ p = 2 $, we recover the sample complexity corresponding to the standard fast rates.\n\n*  **``Q3: The paper focuses on comparing convergence rates rather than absolute final accuracy. Given that FNO has greater expressive capacity, is it possible that with a large enough dataset, FNO might eventually match or outperform the linear estimator?\"**  As discussed above, when the eigenvalues decay polynomially at the rate $ \\lambda_j \\lesssim j^{-p} $ for some $ p > 1 $, the error of our estimator scales as\n$$\n\\lesssim \\frac{||\\mathcal{F} ||\\_{\\text{op}}}{n^{p-1}}.\n$$\nIn contrast, with i.i.d. samples, the best convergence rate achievable by FNO is \n$$\n\\lesssim \\frac{\\text{(some notion of complexity of the FNO model class)}}{n}.\n$$\nFor FNO model to capture $\\mathcal{F}$, the resulting notion of complexity of the model class is generally $\\geq || \\mathcal{F} ||\\_{\\text{op}}$. Thus, while both rates decay to zero as $ n \\to \\infty $, our estimator achieves a faster convergence rate when $ p > 1 $. Therefore, for any given sample size $n$, our estimator should always have a smaller error than FNO in such cases.\n\n*  **``Weaknesses: While theoretically strong, the experimental section is limited to relatively simple PDE cases, and it’s unclear how well the approach scales to more complex, real-world scenarios or nonlinear operators. Some practical implications remain unexplored.\"**  Since our theoretical guarantees apply only to linear PDEs, we focus on standard linear PDEs in our experiments. That said, we agree with the reviewer that extending active learning approaches to nonlinear operators is essential for addressing complex real-world scenarios. We view our work as an important first step that lays a theoretical foundation for future research in that direction.", "claims_and_evidence": "Yes, the paper supports its claims with thorough theoretical analysis and well-designed experiments. It provides detailed proof for the convergence rates of both passive and active learning strategies, clearly demonstrating the advantages of active learning under certain conditions. Additionally, the experimental results on PDE benchmarks align with the theoretical findings, reinforcing the validity of the claims. Overall, the evidence presented is both convincing and comprehensive.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have carefully reviewed the theoretical claims presented up to Section 3.2, including the setup, assumptions, upper bound and the linear estimator, and they appear sound and reasonable to me. However, I find it challenging to fully verify the correctness of the more technical proofs and theoretical results presented in the later sections, particularly those involving detailed operator norm bounds and minimax lower bounds, as they require more advanced mathematical rigor and deeper familiarity with the functional analysis tools used.", "experimental_designs_or_analyses": "Yes, the paper employs both the proposed linear operator estimator and FNO (Fourier Neural Operator) with passive learning as baselines. The experimental design is sound, as it systematically compares these approaches to demonstrate the benefits of active learning. \n\nAdditional Questions to the Authors:\n\nQ1: Could FNO benefit from using the actively selected training data chosen by the proposed linear estimator?\nQ2: The number of training samples in the experiments appears to be chosen empirically. Can they be potentially guided by Theorem 3.1 or any theoretical criteria?\nQ3:  The paper focuses on comparing convergence rates rather than absolute final accuracy. Given that FNO has greater expressive capacity, is it possible that with a large enough dataset, FNO might eventually match or outperform the linear estimator?", "relation_to_broader_scientific_literature": "This paper contributes to the literature on operator learning by providing sharp theoretical comparisons between passive and active learning strategies. It builds upon prior works using Gaussian processes and linear operators, extending them with new minimax optimal error bounds.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-written and offers a clear, rigorous theoretical framework comparing passive and active learning in operator learning. It provides significant insights by establishing minimax optimal error bounds and highlighting when active learning is beneficial. The use of eigenfunctions in the data selection strategy is elegant and well-motivated.\n\nWeaknesses:\nWhile theoretically strong, the experimental section is limited to relatively simple PDE cases, and it’s unclear how well the approach scales to more complex, real-world scenarios or nonlinear operators. Some practical implications remain unexplored.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reBZDbkpKX", "reviewer": "Reviewer_fV81", "summary": "The authors consider active data collection in operator learning with distributions, induced by a stochastic process, over function spaces. They obtained an upper bound for active data collection by spectral techniques and a lower bound for passive data collection, which shows the benefit of the active approach. Concrete examples of various stochastic processes are presented, and experiments are conducted to support the theoretical findings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and for noting that the topic and the problem setting in the paper are intriguing. Below we respond to the main concerns and questions.\n\n*  **On the $L^2$ norm and associated probability measure:**\nWe thank the reviewer for pointing this out. As discussed in Section 2.1, the $L^2$ norm is taken with respect to the base measure $\\nu$. This is usually Lebesgue but can be more general such as Gaussian weighting. We will make this more explicit in the revision to ensure clarity.\n\n* **\"Q1: Provide examples where the assumptions in Section 2.3 apply.\"**  We believe this is a standard framework commonly adopted in operator learning works [1,2]. While Assumption 2.1 is not always explicitly stated, it is typically implicit in the empirical evaluations of these methods. A more detailed discussion of such kernels can be found in [3]. By making this assumption explicit and combining it with active data collection, we are able to derive stronger guarantees—specifically, uniform bound over all distributions in the family $\\mathcal{P}(K)$. In contrast, existing theoretical results usually provide guarantees only for the specific distribution used to generate the training data.\n\n* **Q2: Explain why ``we typically have $\\epsilon \\sim N^{-s}$\" above Section 3: if we consider $\\epsilon$ as an approximation error, the convergence rate here doesn't suffer from the curse of dimensionality. More important, without assumptions on function spaces (just $L^2$ space), a polynomial rate cannot be achieved in operator learning.** \nThe reviewer is correct that achieving polynomial approximation rates is generally non-trivial in operator learning. However, we would like to clarify that $ \\varepsilon $ in our setting does not denote the approximation error of the operator class. Rather, it corresponds to the error in training data due to the error of the PDE solver (i.e., the oracle $ \\mathcal{O} $) used to generate training data.\nThat said, if the reviewer is referring to the error from approximating functions in $ L^2 $ using a truncated basis (e.g., Fourier series), we agree that the commonly stated rate $ N^{-s} $ can be somewhat misleading. For functions defined on a $ d $-dimensional domain, this rate holds only when all Fourier modes $ k \\in \\mathbb{Z}^d $ satisfying $ |k|_{\\infty} \\leq N $ are included. If instead only the first $ N $ modes (in total number) are used, the convergence rate typically becomes $ N^{-s/d} $ for functions with $ s $-degree smoothness. This rate suffers from a curse of dimensionality as the reviewer notes. In our setting, this smoothness assumption is justified: the PDE solver is only applied to the eigenfunctions of the kernel $ K $, which are typically smooth when $ K $ is sufficiently regular. We appreciate the reviewer’s sharp observation and will update the discussion in the final version to clarify this point.\n\n\n[1] Li, Zongyi, et al. ``Fourier Neural Operator for Parametric Partial Differential Equations.\" International Conference on Learning Representations. 2021.\n\n[2] Kovachki, Nikola, et al. \"Neural operator: Learning maps between function spaces with applications to pdes.\" Journal of Machine Learning Research 24.89 (2023): 1-97.\n\n[3] Boullé, Nicolas, and Alex Townsend. \"A mathematical guide to operator learning.\" arXiv preprint arXiv:2312.14688 (2023).", "claims_and_evidence": "The claims are supported by rigorous roofs and experiments.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. Theorem 3.1.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The prior studies focus on both approximation and sampling complexity of operator learning. The active data collection proposed in the paper is to investigate the data sampling and improve statistical efficiency in operator learning. Since operators are defined on more complex topological spaces than Euclidean spaces, the convergence rate is much slower compared to their Euclidean counterparts. The active data collection method propose an interesting idea to improve it.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The topic and the problem setting in the paper are indeed intriguing. However, the theoretical results presented seem quite limited in their mathematical contribution.", "other_comments_or_suggestions": "Given the numerous probability measures discussed in the paper, it is better to specify which probability measure is associated with the L^2 norm to ensure mathematical clarity.", "questions_for_authors": "1. Provide specific examples in operator learning applications which meet the setting in 2.3.\n2. Explain why \"we typically have \\epsilon ~ N^-s\" above Section 3: if we consider \\epsilon as an approximation error, the convergence rate here doesn't suffer from the curse of dimensionality. More important, without assumptions on function spaces (just L^2 space), a polynomial rate cannot be achieved in operator learning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18LbDDexom", "reviewer": "Reviewer_E2pE", "summary": "The authors study the problem of learning a bounded, linear operator through active learning with the assumption that the input functions are drawn from a mean 0 distribution with a known continuous covariance kernel, $K$. \n\nTheir main contribution is a deterministic strategy, which involves solving the Fredholm integral equation to obtain eigenfunctions which are  selected as input functions to query the oracle (eg. a PDE solver). They show polynomial (for fractional inverse), exponential (RBF), 1/n (Brownian motion) convergence rates for some common kernels. \n\nThey motivate their result further with a lower bound of $\\|\\mathcal{F}\\|_{op}^2 \\sum_{j=1}^m \\lambda_j/2$, and show the effectiveness of their method with numerical experiments on the Poisson and Heat Equations by comparing with passive strategies using a linear estimator and Fourier Neural Operators.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comment and noting that our work provides strong theoretical guarantees with empirical evidence. We address reviewer's concern below.\n\n\n*   **``While this is a theoretical study, it would be an interesting next step to see their method on a broader range of PDEs.\"**\n We agree with the reviewer that generating our approach to handle potentially non-linear solution operators is an important future direction. We view our work as an important first step that lays a theoretical foundation for future works on active data collection in operator learning.\n\n*   **``It would be helpful to include an empirical comparison with other active learning methods from the literature, if any exist.\"**\n Currently, there is no widely accepted active learning baseline for operator learning. Unlike the passive setting, where empirical risk minimization on i.i.d. samples is standard, active learning strategies are typically problem-specific and do not scale well to the infinite-dimensional setting.  For example, uncertainty sampling and Bayesian algorithms will both require careful extensions to infinite dimensions.\n\n* **\"It would also be good for the reader to see some intuition for the lower bound construction in the main paper instead of the appendix.\"**\n We will include a short proof sketch of our lower bound construction highlighting key ideas in the main text of the final version of the paper.\n\n* **``Have you considered a randomized sampling method, where input functions could be sampled with probabilities inversely proportional to the Christoffel function? Maybe it could be computationally cheaper?\"**  We thank the reviewer for this thoughtful suggestion. Indeed, computing the top eigenfunctions of the kernel requires solving (or approximating) the Fredholm integral equation, which can be computationally intensive for domains with complex geometry. We had not considered Christoffel function-based randomized sampling, but it seems like a promising approach especially when $d$ is large. We appreciate this idea and will keep it in mind for our ongoing and future works.", "claims_and_evidence": "This paper is mathematically rigorous, providing guarantees for all their results, including theorem 3.1, 4.2 and also showing the measurability conditions to ensure a meaningful definition of $\\mathcal P(\\mathcal X)$, a probability distribution over $L^2(\\mathcal X)$ induced by the stochastic process with kernel $K$.", "methods_and_evaluation_criteria": "Their evaluation framework, of considering error convergence rate in terms of eigen value decay of $K$ makes sense, and experimentally, both the problems they test on Poisson Equation and Heat Equation are relevant applications.", "theoretical_claims": "I did not check the proofs in detail.", "experimental_designs_or_analyses": "Their experiment design sounds well motivated.", "relation_to_broader_scientific_literature": "Their method is in a similar setting as Kovachki et al., 2023, but sample efficient. Their setting is for bounded linear operators, while Lipschitz operators, ones with known SVD has been studied by y de Hoop et al. (2023), Subedi & Tewari (2024), Liu et al. (2024). Their work is a novel active learning method in this specific setting for a known covariance continuous kernel, which goes beyond passive learning. The sample complexity for commonly used kernels used in modeling is also a meaningful advancement.", "essential_references_not_discussed": "I don't think there are.", "other_strengths_and_weaknesses": "Strengths: \n* They provide strong theoretical guarantees, support their method with a lower bound on passive learning, and provide empirical evidence. \n\nWeaknesses: \n* While this is a theoretical study, it would be an interesting next step to see their method on a broader range of PDEs. \n* It would be helpful to include an empirical comparison with other active learning methods from the literature, if any exist.\n* It would also be good for the reader to see some intuition for the lower bound construction in the main paper instead of the appendix.", "other_comments_or_suggestions": "none.", "questions_for_authors": "Your method selects the top eigenfunctions of $K$, and ultimately you leverage the spectral decay of $K$ to achieve fast convergence rates. However it requires exactly solving the Fredholmer integral or numerically approximating it. Have you considered a randomized sampling method, where input functions could be sampled with probabilities inversely proportional to the Christoffel function? Maybe it could be computationally cheaper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myyLrWvqdD", "reviewer": "Reviewer_1DAM", "summary": "The authors study active learning of linear operator motivated by the context of approximating the solution operator of linear PDEs, a setting where the user may \"manufacture\" a small number of queries to give to a PDE oracle in order to approximate the underlying linear solution operator.\n\nIn slightly more detail, the authors consider the problem of learning an infinite dimensional linear operator T over the set of centered, square-integrable stochastic processes with known covariance kernel K arising as the solution function of some PDE $Lu=f$ over domain $X \\subset \\mathbb{R}^d$. The authors work in the (noisy) membership query model $\\mathcal{O}$, meaning they are given access to an oracle $\\mathcal{O}$ which on query a function $u$ over the domain, outputs a function $\\mathcal{O}(u)$ which is within $\\varepsilon$ in L_2 of the true solution $T(u)$. This is meant to model the output of a query made to a numerical PDE solver which may have some small error $\\varepsilon$ based on the discretization level. The learner's goal is to output an operator $\\hat{T}$ minimizing the $L_2$ error from the true operator $T$.\n\n\nThe authors propose a query algorithm for this problem which simply queries the top n eigenvectors $\\{\\phi_i\\}$ of K and outputs the estimator $\\hat{T} = \\sum\\limits_{i=1}^n \\mathcal{O}(\\phi_i) \\otimes \\phi_i$. They prove this estimator has error at most:\n\n$\\varepsilon^2\\sum_{i=1}^n \\lambda_i + ||T||\\sum_{i=n+1}^\\infty \\lambda_i$\n\nwhich is vanishing as the number queries $n \\to \\infty$ and the measurement error $\\varepsilon \\to 0$. \n\nThe authors then explicitly calculate the error rate of their algorithm for several classical settings (fractional inverse of shifted Laplacian, RBF Kernel, Brownian motion...), and show that for many natural parameter settings their active algorithm has significantly improved error rate over passive methods which draw samples from the underlying process rather than actively querying specially constructed functions and typically have at best inverse linear error decay. For instance, for RBF kernels the authors show *exponential* decay of error in 1-D, and better than polynomial decay for any fixed dimension $d$.\n\nThe authors also show that if one only restricts the unknown stochastic process to be generated by some fixed K, there is always an underlying distribution that has *non-vanishing* rate for passive learning, while their active strategy always has rate tending to 0.\n\nFinally, the authors perform a number of experiments showing the advantage of their active procedure over its classical passive counterpart, showing empirical success even in parameter settings where their formal guarantees are not known to hold.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address their concerns below.\n\n*   **``On the assumption of a known covariance kernel $K$:\"** We agree that assuming knowledge of the kernel $K$ may seem unnatural from the perspective of classical statistical learning. However, in operator learning, this assumption is often more reasonable, as data is typically generated synthetically via simulations, where the input distribution is controlled by the user. For example, common covariance operators like $\\alpha(-\\nabla^2 + \\beta I)^{-\\gamma}$ on a periodic domain have known eigenfunctions—specifically, complex exponentials $e^{2\\pi i m \\cdot x}$, with the parameters $(\\alpha, \\beta, \\gamma)$ only affecting the eigenvalues. Since our estimator relies only on the eigenfunctions, it effectively assumes that inputs are represented in a known basis determined by the domain geometry (e.g., Fourier basis on the torus, spherical harmonics on the sphere), not on the detailed spectral properties of the kernel. That said, this separation breaks down for kernels like the RBF (squared exponential), where the scale influences both eigenvalues and eigenfunctions. In such cases, assuming knowledge of the eigenfunctions is indeed more restrictive.\n\n\n* In addition, from a learning-theoretic perspective, assuming knowledge of the kernel $K$ is arguably without loss of generality. In active learning, it's common to assume access to an unlimited pool of unlabeled samples $v_1, \\ldots, v_m \\sim_{\\text{iid}} \\mu$, where $\\mu \\in \\mathcal{P}(K)$, and focus on minimizing label complexity—the number of labeled samples requested (see Hanneke, 2014). This aligns with our setting, where labeling (e.g., solving a PDE) is the primary cost. Given such unlabeled samples, one can estimate the covariance operator as:\n$$ \\Sigma\\_m = \\frac{1}{m-1} \\sum_{i=1}^m (v_i - \\overline{v}\\_m) \\otimes (v_i - \\overline{v}\\_m),  $$\nwhere $\\overline{v}\\_m = \\frac{1}{m} \\sum_{i=1}^m v_i.$ \nSince $\\mathbb{E}[||v_i||^2] < \\infty$, Theorem 8.1.2 of Hsing and Eubank (2015) guarantees that $\\Sigma_m \\to \\Sigma$ almost surely in Hilbert-Schmidt norm, where $\\Sigma$ is the integral operator associated with $K$. While our work assumes $\\Sigma$ has a finite trace norm, this is not required to recover its eigenfunctions: convergence in Hilbert-Schmidt norm suffices for accurate top $n$ eigenfunctions approximation. Thus, assuming access to the eigenfunctions of $K$ is reasonable in theory, even if it may be computationally demanding in practice. We also note that this does not contradict known lower bounds in pool-based active learning, since our approach queries the oracle on estimated eigenfunctions of $\\Sigma_m$, which are not i.i.d. samples from $\\mu$.\n\n\n* **``Can one show knowledge of $K$ is necessary to achieve fast rates, at least in some settings?\"**\nYes, some knowledge of the kernel $K$ is necessary to achieve fast rates for certain estimators. Earlier, we showed that if the learner has access to unlabeled samples from a distribution $\\mu \\in \\mathcal{P}(K)$, then $K$ can be estimated from data, making the assumption of known $K$ natural. Here, we consider the more adversarial setting where the learner has no access to such samples. Suppose the learner selects $n$ inputs $v_1, \\ldots, v_n$ using any active strategy and receives exact labels $w_i = \\mathcal{F}(v_i)$. Let $\\widehat{\\mathcal{F}}\\_n$ be the resulting estimator. As is common with many linear estimators, we assume $\\widehat{\\mathcal{F}}\\_n(v) = 0$ for any $v$ outside the span of $v_1, \\ldots, v_n$. This models abstention outside the observed subspace. Since learner had no knowledge of $K$, we can construct a kernel\n$$\nK(x, y) = \\sum_{j=1}^M \\lambda_j \\varphi_j(x) \\varphi_j(y),\n$$\nwhere the orthonormal functions $\\varphi_j$ are chosen to be orthogonal to the span of the learner's queries. This is always possible in infinite-dimensional $L^2(\\mathcal{X})$. We then define $\\mu \\in \\mathcal{P}(K)$ as the law of a Gaussian process with kernel $K$.\nSince the entire support of $\\mu$ lies outside the learner’s observed subspace, the estimator cannot generalize. In particular,\n$$\n\\mathbb{E}[||\\widehat{\\mathcal{F}}\\_n(v) - \\mathcal{F}(v)||^2] \\geq \\sum_{j=1}^M \\lambda_j ||\\mathcal{F}(\\varphi_j)||^2.\n$$\nIf $\\mathcal{F}$ is not finite-rank, we can always find some $\\varphi_\\ell$ with $||\\mathcal{F}(\\varphi_\\ell)|||^2 \\geq c > 0$. By setting $\\lambda_\\ell = 1$ and choosing the rest of the $\\lambda_j$ to satisfy a trace constraint, the lower bound remains constant, independent of $n$. In short, if the learner lacks both knowledge of $K$ and access to samples from $\\mu \\in \\mathcal{P}(K)$, there exist kernels that force all probability mass outside the learner’s span, leading to a non-vanishing error regardless of how data is collected.\n\n\n[1] Hanneke, Steve.``Theory of active learning.\" Foundations and Trends in Machine Learning 7.2-3 (2014).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proof sketch provided in the main body is convincing of their main theorem.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to establish *active learning* rates for linear operator estimation. Prior work focused only on the passive setting, but as the authors reasonably argue, in many cases we have access to an approximate PDE solver which we can feed queries of our own design. They give a very simple query algorithm achieving substantially better error rates than can be achieved passively so long as one can compute the eigenfunctions of the known covariance kernel.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "The membership query model is very infrequently studied in active learning due to generally being considered unrealistic (e.g. one cannot really give a synthesized data point to a human being to label and expect a coherent response). One additional strength of this work is identifying (linear) PDE solving as a potential application where the membership query model really is plausible, and provides substantially improved error rates over its passive counterpart (or in fact even over the standard active pool-based model, which I believe the authors lower bound also holds for).\n\nFrom a learning standpoint, it is a bit disappointing that the proposed algorithm relies so strongly on knowing the covariance kernel K, and I am not convinced the assumption is all that reasonable (though the authors make the case it is a standard assumption in prior work on operator learning). Knowing K allows one to work directly with its (known) eigenfunctions, sort of immediately unlocking \"PCA\" style techniques with no need for estimation. Often in statistical learning one would expect to have to learn some of the underlying geometry to do something like this which is avoided here.\n\nEDIT: I am largely happy with the authors' response regarding known vs unknown K, and suggest they include a formalization of the discussion in the next version.", "other_comments_or_suggestions": "GP denotes Gaussian Process is stated well after the first time GP is used.", "questions_for_authors": "Can one show knowledge of K is necessary to achieve fast rates, at least in some settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m99zmVgmh8", "reviewer": "Reviewer_rbqN", "summary": "This paper studies the approximating solution operator of the PDE using active learning queries given the kernal. The paper includes an upperbound for an active learning setting that converges to an irreducible error with a larger number of queries. Further, the authors show a lower bound for passive learning. The paper also provides some numerical experiments using their estimators to learn Poisson and Heat Equations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting that our work provided rigorous proofs and numerical experiments.\n\n* **“I wonder if the authors can discuss what other types of queries they think might be interesting for PDE approximation if they have any in mind.”**  \nOne natural direction would be to explore a pool-based active learning setting, where the learner selects inputs from a large (possibly infinite) unlabeled pool drawn from a fixed distribution $ \\mu $, rather than querying arbitrary inputs. This may be more appropriate in scenarios where labeling requires running physical experiments, making arbitrary queries impractical. However, in the context of PDE surrogate modeling, where data is generated synthetically, we believe the membership-type query model we adopt that allows arbitrary input queries is reasonable.", "claims_and_evidence": "The submission provides both rigorous proof and numerical experiments.", "methods_and_evaluation_criteria": "Both the problem setting and the strategy of solving Feldhome integral for picking queries make sense.", "theoretical_claims": "I went through the proof for Theorem 3.1 in Appendix A. I did not find any issues.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "This paper proposed a new setting for operative learning and provided an algorithm with theoretical guarantees.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The authors used active learning with membership queries for the setting, which seems most natural. However, I wonder if the authors can discuss what other types of queries they think might be interesting for PDE approximation if they have any in mind.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the mathematical theory of learning linear operators via active learning. Bound on the excess generalization error is derived in terms of the eigenvalues of the input process covariance operator. Lower bound for passive learning is also derived. Only the proof for Theorem 3.1 in Appendix A was checked by Reviewer rbqN. Minor concerns raised focus on insufficient numerical experiments (Reviews XHLU, UK7y, fV81, E2pE), insufficient details of experiments (Reviews XHLU), incremental theoretical contribution (Review fV81), and theory relies on knowing the eigenvalues of covariance kernel (Review 1DAM).
Decision: Accept (spotlight poster)