Ensemble and Mixture-of-Experts DeepONets For Operator Learning
Authors: Ramansh Sharma
Varun Shankar
Venue: Accepted by TMLR ()

Abstract: We present a novel deep operator network (DeepONet) architecture for operator learning, the ensemble DeepONet, that allows for enriching the trunk network of a single DeepONet with multiple distinct trunk networks. This trunk enrichment allows for greater expressivity and generalization capabilities over a range of operator learning problems. We also present a spatial mixture-of-experts (MoE) DeepONet trunk network architecture that utilizes a partition-of-unity (PoU) approximation to promote spatial locality and model sparsity in the operator learning problem. We first prove that both the ensemble and PoU-MoE DeepONets are universal approximators. We then demonstrate that ensemble DeepONets containing a trunk ensemble of a standard trunk, the PoU-MoE trunk, and/or a proper orthogonal decomposition (POD) trunk can achieve 2-4x lower relative $\ell_2$ errors than standard DeepONets and POD-DeepONets on both standard and challenging new operator learning problems involving partial differential equations (PDEs) in two and three dimensions. Our new PoU-MoE formulation provides a natural way to incorporate spatial locality and model sparsity into any neural network architecture, while our new ensemble DeepONet provides a powerful and general framework for incorporating basis enrichment in scientific machine learning architectures for operator learning.

Published in Transactions on Machine Learning Research (03/2025)
Ensemble and Mixture-of-Experts Nes For Operator Learning
Ramansh Sharma
Kahlert School of Computing, University of Utah, UT, USA ramansh0cs. utah.eda edu
Varun Shankar
Kahlert School of Computing, University of Utah, UT, USA shankar@cs. utah. edu
Reviewed OpenReview:
Abstract
We present a novel deep operator network (DeepONet) architecture for operator learning, the ensemble DeepONet, that allows for emriching the trunk network of a single DeepONet with multiple distinct trunk networks. This trunk enrichment allows for greater expressivity and generalization capabilities over a range of operator learning problems. We also present a spatial mixture-of-cxperts (MoE) DeepONet trunk network architccture that utilizes partition-of-unity (PoU) approximation to promote spatial locality and model sparsity in the operator learning problem. We first prove that both the ensemble and PoU-MoE DeepONets are universal approximators. We then demonstrate that ensemble DeepONets containing a trunk ensemble of a standard trunk, the PoU-MoE trunk, and/or a proper ort hogonnll decomposit (POD) trunk can achieve 2-4x lower relative & errors Ihan standard DeepONets and Nets on both standard and challenging new operator learning problems involving partial differential equations (PDEs) in two and three dimensions. Our new PoU-MoE formulation provides natural way to incorporale spatial locality and model sparsity into any neural network architecture, while our ensemble DeepONet, provides powerful and general framework for incorporating basis enrichment in scientific machine learning architectures for operator learning.
1 Introduction
In recent. years, machine learning (ML.) has been applied with great success to problems in science and engineering. Notably, ML architectures have been leveraged to learn operators, which are cunetion-to-functeo maps. In many of these applications, ML-based operators, often called newral operators, have been utilized to learn solution maps to partial differential equations (PDEs). This area of research, known as operator learning, has shown immense potential and practical applicability variety of real-world problems such as weather/clinge /elimate modeling (Bora al. 2023 Pathak al.) 2022), earthquake modeling (Haghighat al.,
2024) material science (Gupta & Brandstetter 2022 Oommen al. 2023 and shape optimization (Shukla et al.
)024). Some popular neural operators that have emerged are deep operator networks (DeepONets) (Lu 2021.. Fourier neural operators (FNOs) (Li et al. 2021 graph neural operators (GNOs) (Li et al. 2020 and more recently convolutional neural operators (CNOs) Raonic el al. (2023). DeepONets have also been extended to incorporate discretization invariance (Zhang et al. 2023), more general mappings (Jin et al.) 2022), and multiscale modeling (Howard et al. (023). In this work, we focus on the DeepONet architecture due its ability to separate the function spaces involved in operator learning: for completeness, we discuss possible extension to the FNO in Appendix B
ct al.

---

Published in Transactions on Machine Learning Research (03/2025)
At high level, operator learning consists of learning map from an imput function to output function. The DeepONet architecture is an inner product between a trunk network that is a function of the output function domain, and a branch network that learns to combine elements of the trunk using transformations of the input function. In fact, one can view the trunk as a set of learned, nonlincar, data-dopendent basis functions. This perspective was first leveraged lo replace the trunk with a set of basis functions learned from a proper orthogonal decomposition (POD) of the training data corresponding to the output functions; the resulting POD-DeepONet achieved state-of-the-art accuracy on a variety of operator learning problems (Lul et al., 2022). More recently, this idea was further generalized by extracting a basis from the trunk a postprocessing step (Lee & Shin 2023): this approach proved to be highly successful in learning challenging operators PPeyvnn et al. 2024).
In this work, we present the ensemble DeepONet, a DeepONet architecture that explicitly enables enriching a trunk network with multiple distinct trunk networks; however, this /angene//nggmeneed trunk uses single branch that learns how to combine multiple trunks in such a way as to minimize the DeepONet loss function. The ensemble DeepONet essentially provides a natural framework for busis function enrichment of a standard (vanilla) DeepONet trunk. We also introduce novel partition-of-un of-unity (PoU) mixture-of-experts (MoE) trunk, the PoU-MoE trunk, that produces smooth blends of spatially-localized, overlapping, distinct trunks. The use of compactly-supported blending functions allows the PoU formulation to have a strong inductive bias towards spatial locality. Acknowledging that such an inductive bias is not always appropriate for learning inherently global operators, we simply introduce this PoU-MoE trunk into our ensemble DeepONet as an ensemble member alongside other global bases such as the POD trunk.
Our results show that the ensemble DeepONet, especially the POD-PoU ensemble, shows 2-4x accuracy improvements over vanilla-DeepOO DeepONes with single branches and up to 2x accuracy improvements over the POD DeepONet (also with a single branch) in challenging 2D and 3D problems where the output function space of the operator has functions with sharp spatial gradients. In Section strengths of five different ensemble formulations, cach carcfully selected to answer a specific scientific question about the effectiveness of ensemble DeepONets. We conclude that the strength of ensemble DeepONets lie not merely in overparametrization but rather in the ability to incorporate spatially local information into the basis functions.
! we summarize the relative
1.1 Related work
Basis enrichment has been widely used in the field of scientific computing in the extended finite element method (XFEM) (MeQuien et al.] 2020; Belytschko & Black 1999; Ballard et al. 2022). modern radial basis
function (RBF) methods (Flyer et al. 2016 Bayona et al. 2019 Shankar & Fogelson 2018 Shankar et al., 2021), and others (Cai et al.) 2001). In operator learning, basis enrichment (labeled "feature expansion" with trigonometric functions was leveraged to enhance accuracy in DeepONets and FNOs (Lu et al. 2022) The ensemble DeepONet generalizes these prior results by providing a natural framework to bring data-dependent, locality-aware, basis function enrichment into operator learning. PoU approximation also has a rich history in scientific computing (Melenk & Babuvska
1996
Larsson et al., 2017 (Shcherbkkov & Larsson 2016
Heryudono et al.] 2016; VSaflari-Viihani et al. 2015
Shankar & Wright 2018), and has recently found use
in ML applications (Han et al.) 2023 Cavoretto et al. 2021 Trask et al. 2022). In (Trask et al. 2022
which targeted (probabilistic) regression applications, the authors used trainable partition functions that were effectively black-box ML classifiers with polynomial approximation on each partition. In Han et al. (0023)
(which also targeted regression), the authors used compactly-supported kernels as weight functions (like in this work), but used kernel-based regressors on each partition. Our PoU-MoF. formulation generalizes both these works by using ncural networks on each partition and further generalizes the technique to operator learning. In general, ensemble learning and MoE have rich history, and we provide a more in-depth overview in Appendix A The ensemble and PoU-MoE DeepONets introduced here extend this body of work to deterministic operator learning and PDE applications.
Broader Impacts: To the best of the authors' knowledge, there are no negative societal impacts of our work including potential malicious or unintended uses, environmental impact, security, or privacy concerns. The ensemble-DeepOX. proposed in this work using the POU-MoE. mothod have higher training times than their bascline counterparts and as such warrant consideration of the computational cost when being used in practice.
2

---

Published in Transactions on Machine Learning Research (03/2025)
u
Branch network
9 (u, y)
PU
Pu (y), "PU (V),
(y) Pu
y
I POD]
[v)()), 01(y), ... Opp 119)
POD
Ensemble POD-PoU trunk
Figure 1: An ensemble DeepONct containing a POD trunk and a PoU-MoE trunk.
Limitations: Ensemble DeepONets, especially when using PoU-MoE trunks, contain 2-3x many trainable trunk network parameters vanilla-DeepONet and consequently require more time to train (see Section 3.4 for runtime results and discussion); however, in future work, we plan to ameliorate this issuc with a novel parallelization strategy for the PoU-MoE trunk. Further, due to limited time, we used a single branch network that outputs to Rº for all our results (an unstacked branch) rather than using p branch networks that each output to I (a stacked branch) from Lu et al. (2022). This choice may result in lowered accuracy for all methods (not just ours), but certainly resulted in fewer paramcters Lu al. 2022 However, our results exlend straightforwardly to stacked branches also.
2 Ensemble DeepONets
In this section, we first discuss the operator learning problem, then present the ensemble DeepONet architecture for learning these operators. We also present the novel PoU-MoF, trunk and a modification the POD trunk from the POD-DeepONet, both for use within the ensemble DeepONet.
2.1 Operator learning with DeepONets
Let u (Su, I and V (2,; be two separable Banach spaces of functions taking values in Q2 C RR and a general (nonlinear) operator. The operator learning
22 C Rdd,, respectively. Further, 9 + problem involves approximating 9 + V with a parametrized operator 9 1 - from a finite number of function pairs {(u,, u.),, i ,,...,N N where us U are typically called imput functions, and e e are called output functions, L..,, 84 = (u,)). The parameters O are chosen to minimize ||g タll in some norm.
In practice, the problem must be discretized. First, one puts samples the input and output functions at a finite set of function sample locations X E Ωg and Y E Pw, respectively; also let N = =X| and N, = |Y). One then requires that [|vı(y) - (v,))))) is minimized over (w;, vi), i == 1, I,...,,, N, where u, are sampled at X and es at y E Y. The vanilla- Deella--eeeeNett is one particular parametrization of (()(()) (()(v)) = (T(), Buu)) + bu where (. is the p-dimensional inner product, B R. x O2 + R is the branch (neural) netw + RRd x 0 - I2 is the trunk nerwork, and bo is a trainable bias parameter; is a hyperparameter that partly controls the expressivity (v). Og and O are the trainable parameters in the branch and trunk, respoctively.
2.2 Mathematical formulation
We now present the new ensemble DeepONet formulation; an example is illustrated in Figure 1 Without loss of generality, assume that we are given three distinct trunk networks Ti (1; (,2 0r and (3 0r,), where y corresponds to the domain of the output function v(y). Assume further that Tj : Rd x O,, - R?; 1,2,33 Then, given single branch network B(u: 0b), the ensemble DecpONet is given in vector form by:
((w)(v) = (171 0 T2 0r.) T2(u; (
by -00-((())))) (.: + by-
(1)
Here, +: dd 0r, x 172 Or, Ro tro is the ensemble trunk. Clearly, the individual trunks simply "stack" column-wise to form the ensemble trunk 9; in Appendix C we discuss other suboptimal attempts
3

---

Published in Transactions on Machine Learning Research (03/2025)
A
B
C
Spattall Gradient
Poll basis
basis
0.1
0.5
0.65
Enriched bases on the 2D reaction-diffusion problem 3.2 The solutions exhibit sharp gradients (left); the PoU-MoE trunk has learned spatially-localized basis functions (middlo); the POD trunk has learned a global basis function (right).
Figure 2:
to form ensemble trunk. The ensemble trunk now consists of P1 + P2 + P3 (potentially trainable) basis functions, necessitating that the branch B R. Og peitretps
A universal approximation theorem
Theorem 1. Let G U + V be a continuous operator. Define & as ((u)(v) = - 0r : 0rs; ((u; 0b)) +
bo, where B No 0 + patps is branch network embedding the input function u, bo is the bias, and 4 :d θz, 0r, O t is ensemble trunk network. Then, for any O, there erists ; that con approcimale G globally such that
(V)( (u
(2)
where >> O be made orbilrarily small and Ily is the norm of the Banach space V.
Proof. This automatically follows from the (generalized) universal approximation theorem (Lu et al. which holds for arbitrary branches and trunks.
2021
2.2.1
The PoU-MoE trunk
We now present the PoU-MoE trunk architecture, which leverages partition-of-umity approximation. We begin by partitioning Q, into P overlapping circular/spherias patches Ωu. k l...., P, with each patch
having its own radius Pk and containing set of sample locations Ys: of course, 0 Yv Y. The key idea
behind the PoU-MoE trunk is to employ separate trunk network on each patch Qs and then blend (and train) these trunks appropriately to yield single trunk network on Ω. Each T is trained at data on Ys. but may also be influenced by spatial neighbors. The PoU-MoE trunk Tru (x) is given as follows:
Tee (v: O OPP
Σ (u.
3
where 0 A = P are the trainable parameters for each trunk. In this work, choose the weight functions We to be (scaled and shifted) compactly-supporte positive-defintee kernels th Rd x Rd + R that
are C R2d) More specifically, on the patch Ω4 we select the to be the C² (RR²) Wendland kernel (Wendland Fasshaucr & McCourt) 2015), which is a radial kernel given by
1995
2005
Fasshauer
2007
do (v. v 2/k 11y vell vkll) (h)(r) (1-r) (1-r) 0.
if if 1
(4)
where of the center of the k-th patch. The weight functions are then given by
vx(())
wx(x) = E, ())) k.j = I. P.
(5)
4

---

Published in Transactions on Machine Learning Research (03/2025)
which automatically satisfy Es wx(y) = 1. Each trunk Tk can be viewed an "expert" on its own patch Re, thus leading to spotial MoF. formulation via the PoU formalism. Both training and evaluation of Tpu can proceed locally in that each location y lies in only a few patches; our implementation leverages this fact for efficiency. Further, since the weight functions wx(y) are each compactly-supportee on their own patches 22g Te can be viewed as sporse in its constituent spatial experts тк- Nevertheless, by ensuring that neighboring patches overlap sufficiently, we ensure that Te still constitutes global set of basis functions. For simplicity, we use the same p value within each local trunk Tk- Figure 2 (middle) shows one of the learned PoU-MoE basis functions in the POD-PoU ensemble; the learned basis function exhibits strong spatial locality corresponding to partitions. In Section 3.2 we present more evidence for this spatial localization in the PoU-MoE basis functions.
Partitioning: We placed the patch centers in bounding box around Ω place Cartesian grid in that box, then simply select P of the grid points to use as centers. In this case, the uniform radius is determined as (Larsson et. al. 2017) p (1 + 5)0.5 v where ō is a free parameter to describe the overlap between patches and H is the side length the bounding box. However, as a demonstration, we also used variable radii Pk in Section 3.1 In this work, we placed patches by using spatial gradients of a vanilla-DeepONee as our guidance, attempting to balance covering the whole domain with resolving these gradients; see Appendix E.3 for more in-depth discussion on partitioning strategies.
A universal approximation theorem Theorem 2. Let g
U
be a continuous operator.
Define Gt as (l)
B(u; 0o), Σ [uy(t),,((i00))
+ bo, where R:R. 0g - RP is branch network embedding the input
function u, Tj : Rd Hr, + Rp are trunk networks, by is bias, and wj Rd. + R are compactly-suuportedl posilive-defiree definile weight functions that salisfy the partition of unity condition Ej wz(() = 1, = I,..., P. Then, for any c <>0 0, there exists Gt that can approzimate 9 globally such that
(Q(()())) (w)
(6)
where& > 0 can be made arbitrarily small and || Ily is the norm of the Banach space V.
Proof. See Appendix D for the proof. The high level idea is to use the fact that the (generalized) universal
approximation theorem (Chen & Chen 1995: Lu et al. 2021 already holds for each local trunk on a patch, then use the partition of unity property to effectively blend that result over all patches to obtain a global estimate.
2.2.2 The POD trunk
The POD trunk is modified version of the trunk used in the POD-DeepONet of the output function data. First, we remind the reader of the POD procedure |Chatterjce 2000). Recalling that (v) are the output functions, first define the matrix ViJ = 1 a (v,(,)) - n.), where 14 is the spatial mean of the i-th function and o; is its spatial standard deviation. Define the matrix T N vVv, and let & be the matrix of eigenvectors of T ordered from the smallest eigenvalue to the largest. Then, the POD-DeepONet involves P. selecting the first p columns of & to be the trunk of a DeepONet so that GroD (w)()) = Σ (,(w), + do(v),
where (y) is the mean function of v(y) computed from the training dataset, and (v()) are the columns of & explained above. In this work, use a POD trunk that includes the mean function de in the set of basis functions. We label this the "Modified-POD" trunk in our experiments; this "Modified-POD" trunk FoD is given by
Tron (v) [do(s) Φ1 (v)
op-1(y)).
Consistent with the POD-DeepONet philosophy, no activation function is needed and the POD trunk has no trainable parameters. Figure 2 (right) shows one of the learned POD basis functions in the POD-PoU ensemble.
5

---

Published in Transactions on Machine Learning Research (03/2025)
2.2.3 Other Neural Operators
While we restricted attention to DeepONets in this work, the ensemble idea naturally extends to other neural operator architectures. In Appendix B we briefly discuss our ideas creating ensembles of global and local basis functions within the FNO.
3 Results
We present results of our comparison of the new ensemble DeepONet (with and without PoU-MoE trunk) against vanilla and POD DeepONets. We considered different ensemble combinations of the vanilla, POD, and PoU-MoF trunks. Each of the following ensembles attempted to address a specific scientific question:
1. Vanilla-POD: Does adding POD modes to a vanilla trunk enhance expressivity over using cither trunk in isolation?
2. Vanilla-PoU: Does spatial locality introduced by the PoU-MoE trunk aid DeepONet?
3. POD-PoU: Does having both POD global modes and PoU-MoE local expertise enhance expressivity simply using vanilla trunk?
4. Yanilla-POD-POU:: If the answer above is affirmative, then does adding a vanilla trunk (representing extra trainable parameters) to POD-PoU ensemble help further enhance expressivity?
1I+ Is spatial localization truly important or is simple overparametrization all that is needed? We use P 1 vanilla trunks in this model, where P is the number of PoU-MoE patches. This ensemble thus contains many trunks as the vanilla-PoU or POD-PoU ensembles, but all basis functions are purely global in this setting.
The answers to these questions are shown in Table and summarized in Section 4 In nutshell, spatial localization is indeed important, as is using a mix of global and localized basis functions; simple overparametrization is insufficient to attain of accuracy. We now describe our experimental setup, and both the standard and novel benchmark test results that led us to this conclusion.
Important DeepONet details. In all cases, for parsimony in the number of training parameters, we used a single branch (the unstacked DeepONet) that outputs to R' rather than p branches. We found that output normalization did not help siguificantly in this case, We scaled all our POD architecture outputs by 1 (standalone or in ensembles), as advocated in Lu el al. 2022
Experiment design. In the remainder of this section, we establish the performance of ensemble DeepONets on benchmarks such as a 2D lid-driven cavity flow problem (Section 3.1 and 2D Darcy flow problem on a triangle (Appendix F.1 1 both common in the literature (Lu el al. 2022: Batlle et al. 2024). However, we also wished to develop challenging new spacetime PDE benchmarks where the PDE solutions (output functions) possessed steep gradients, while the input functions were well-behaved. To this end, we present results for both 2D reaction-diffusion problem (Section 3.2) and 3D reaction-diffusion problem with sharply (spatially) varying diffuuion coefficients (Section 3.3 In both cases, constructed spatially discontinuous reaction terms that resulted in PDE solutions (output functions) with steep gradients. Such PDE solutions abound in scientific applications. We note at the outset that the ensemble DeepONet with the PoU-MoE trunk performed best when the solutions had steep spatial gradients. Results on the Darcy problem show that the ensemble approaches tested here were not as effective on that problem.
Error calculations. For all problems, we compared the vanilla- and POD-DeepONets with the five different ensemble architectures described at the top of Section 3 We also compared these ensembles against a DeepONet with the modified POD trunk from Section 2.2.2 (labeled Modifted-POD), For all experiments, we first computed the relative 12 error for each test function, Ct2- = [R-ulla where was the true solution kxll
vector and was the DeepONet prediction vector; we then computed the mean over those relative & errors. For vector-valued functions, we first computed pointwise magnitudes of the vectors, then repeated the same process. The MSE loss was used to train all the models. We also report squared error (MSE) between the DeepONet prediction and the true solution averaged over N functions emse (y) N (ū(v) =²

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Relative la errors (as percentage) on the test dataset for the 2D Darcy flow, cavity flow, and reaction-diffusion, and the 3D reaction-diffusin problems. RD stands for reaction-diffssin.
Darcy flow
Cavity flow
2D RD
3D RD
Vanilla POD
0.857 + 0.08
5.53 : 1.05
0.144 0.01
0.127 + 0.03
0.297 + 0.01
7.94 + 2e -
5.06 + -
9.40 8 8
Modified-POD
+.004
2e -
0.131 4
0.155 -
(Vanilla, POD)
0.227
0.310
0.0751 4e
5.24 10.4
(P+ + Va-iilllla
1.19 0.06
2.17
0.0644 0.02
5
(Vanilla, PoU)
0.976
1.06
0.0946 0.03
5.25 10.3
(POD, PoU)
0.204 0.02
0.204 0.01
0.0539 4e 5
0.0576 0.05
(Vanilla, POD, PoU)
0.187 = 0.02
0.229 + 0.01
0.0666 + 8e 5
5.22 - 10.4
Notation. In the following text, we denote the space and time domains with Q and T respectively; the spatial domain boundary is denoted by On. A single spatial point is denoted by y, which can either be a point 82) in R2 or point (y1, V2. y3) in R
Setup. We trained all models for 150,000 epochs on an NVIDIA GTX 4080 GPU. All results were calculated over five random soeds. We annealed the learning rates with an inverso-time decay schedule. We used the Adam optimizer (Kingma & Ba, 2017 for training on the Darcy flow and the cavity flow problems, and the AdamW optimizer (Loshchilov & Hutter 2018) on the 2D and 3D reaction-diffusion problems. Other DeepONet hyperparameters and the network architectures are listed in Appendix B
3.1 2D Lid-driven Cavity Flow
Input function
Output (a)
partitioning
F1
Fi
Vanilla
(Vanilla, POD, PoU) aec
(POD, PoU)
g
y1
Figure 3: The 2D lid-driven cavity flow problem. We show in (A) an example input function; in (B) an example output function component; in (C) the four patches used for the PoU-MoE trunk; in (D), (E), and (F) the spatial mean squared error (MSE) for the vanilla, ensemble Manila-POO-Per and ensemble POD-PoU DeepONets respectively (see Error calculations in Section for computing MSE for vector-valued functions).
The 2D lid-driven cavity flow problem involves solving for fluid flow in a container whose lid moves tangentially along the top boundary. This can be described by the incompressible Navier-Stokes equations (with boundary

---

Published in Transactions on Machine Learning Research (03/2025)
conditions),
Ōu
(u Vju Vp + vAu, V.u Ve tET,
(8)
Ōt
u Ub
(9)
where u = (a(y), v(y) the velocity field, p is the pressure field, v is the kinematic viscosity, and u, (ug, Ub) is the Dirichlet boundary condition. We focused on the steady state problem and used the dataset specified in Lu [2022] Section 5.7, Case A). We set L [0, 1J2 and learned the operator g u. - The steady state boundary condition defined as,
ub U cosh (r(x-2))
21) (11--o0 cosh (5)
(10)
tb
where r 10. The other boundary velocities were set to zero. described Lu et al. (2022), the equations were then solved using lattice Boltzmann method (LBM) generate 100 training and 10 input and output function pairs. All function pairs generated over range of Reynolds numbers in the range [100, 2080] (with U and chosen appropriately), with no overlap between the training and test dataset. Figure 3 shows the four patches used to partition the domain.
We report the relative & errors (as percentage) on the test dataset in Table 1 The vanilla-, modified POD-, and POD--DepPNNets had the highest errors (in increasing order). The POD-PoU ensemble was the most accurate model by about an order of magnitude over the Venilla-DeepONel, and almost two orders of magnitude over the POD variants. While all ensembles outperformed the standalone DeepONets, the ensembles possessing POD modes appeared do best general. Further, adding a PoU-MoE trunk to the ensemble seemed aid accuracy in general, but especially when POD modes were present. The spatial MSE figures in Figure 3 reflect the same trends.
3.2 A 2D Reaction- Difffusii Problem
A
B
C
Input function
Output function
PoU partitioning
km
MC4
D
Vanilla
(P+1)-Vanlla
(POD, PoU)
g.
50
Y1
F1
Figure 4: The 2D reaction-diffusion problem. We show in (A) an example input function; in (B) an example output function: in (C) the six patches used for the PoU-MoE trunk; in (D), (E), and (F) the spatial mean squared error (MSE) for the vanilla, ensemble (P + 1)-vanilla, and ensemble POD-PoU DecpONets respectively.
8

---

Published in Transactions on Machine Learning Research (03/2025)
basis
0.35
0.35
1.5
0.25
0.25
8.
0.20
0.35
9.10
0.10
0.00
0.00
y1
F1
Figure 5: Vanilla-eeepONeet and PoU (from POD-PoU ensemble DeepONet) basis functions for the largest. branch modes on the 2D cifftionddiffusion problem.
Nexl, we present experimental results on 2D reaction-diffusion problem. This equation governs the behavior of chemical whose concentration is c(y, t), and is given (along with boundary conditions) below:
dc Kon (R- c) Camb - kof c + vdc, y E S2 tET,
(11)
Ōt
with the boundary condition 3 0 on A. The first r.h.s term is binding reaction term modulated by kon
and the second term an unbinding term modulated by koff- Camb Camb f)
cos(2 cos(2Ty2)) exp(-at) is
background source of chemical available for reaction, p = 0.1 is the diffusion coefficient, R = 2 is a throttling term, and n(y) is the unit outward normal vector on the boundary. In our experiments, we used Ω = [0, 2
and T" = [0, 0.5]. We set the initial condition as a spatial constant c(y, 0) U(0, 1). More importantly, kon and kof are discontinuous and given by
2, 11 S 1.0, 0, otherwise
0.2, S 1.0,
kon
(12)
0, otherwise
where is the horizontal direction. This discontinuity induces sharp solution gradient at y1 = 1.0 (see Figure 4 (B)). Our goal was to learn the solution operator G c(v,0) + c(y, 0.5). We solved the PDE numerically at Ny = 2207 collocation points using a fourth-order accurate RBF-FD method (Shankar & Fogelson 2018 Shankar al. 2021); using this solver, we generated 1000 training and 200 test input and output function pairs. We sampled the random spatially-constant input on regular spatial grid for the branch input. We used six patches for the PoU trunks as shown in Figure 4
The third column of Table [ shows that the POD-PoU ensemble achieved the lowest error, with an error reduction of almost 3x over the standalone DecpONets. The (P + 1)-vanilla ensemble also performed reasonably well, with a greater Ihan 2x error reduction over the same; this indicates that eerparameerizatiion indeed helped on this test case. However, the relatively higher errors of the vanilla-PoU ensemble (compared the best results) indicate that POD modes are possibly vital fully realizing the benefits of the PoU-MoE trunk. Once again, the spatial MSE plots in Figure corroborate the relative errors.
Spatial Localization of the PoU-MoE Basis: We present further evidence showing that the PoU-MoE trunk learns spatially local fcatures. In Figure 5 we show basis functions from the vanilla-eeepONet trunk and the PoU-MoE trunk of the POD-PoU ensemble DeepONet. Unlike the basis functions shown in Figure 2 these correspond to the largest branch coefficients in the respective models, i.e., the most "important" basis functions. Clearly the PoU basis has significantly higher spatial variation than the vanilla basis. We believe that this learned spatial locality helps the ensemble DeepONets with the PoU-MoE trunk achieve superior accuracy problems with strong local features (such those tested in this work).
3.3 3D Reaction- Sraction-Variable-.o Corfabbe-Coefiicient Diffusion
Finally, we present results on 3D paction-diffusion problem with variable-coefficte diffusion. We used similar setup to the 2D but significantly also allow the diffusion coefficient. to vary spatially via.
9

---

Published in Transactions on Machine Learning Research (03/2025)
PoU partitioning
Input function
Output function
C
A
14
15
1.
25
6.s M
0.6
4.
12
X1
Y1
y1
Vanilla
Modified POD
(POD, PoU)
1e-6
1e-B
18
40
4.5
45
91
4.5
Figure The 3D reaftion-iiffusion problem. We show in (A) an example input function; in (B) an example output function; in (C) the eight patches used for the PoU-MoE trunk; in (D), (E), and (F) the spatial mean squared error (MSE) for the vanilla, modified POD, and ensemble POD-PoU DeepONets respectively.
lunction K(y), y E R3. The PDE and boundary conditions are given by dc Ot
kon (R - c) Camb kof c V (K())Ve), VER,IET,
(13)
with K(y) dc = O on a.. Here, Ω was the unit ball, i.e., the interior of the unit sphere S², and T = [0, 0.5).
8n We set the kon and hioff coefficients to the same values as in 2D in S O, and to zero in the y1 > 0 half of the domain. We set Camb = (1 + cos(2 cos(2 sin((rys))e ( All other model parameters were kept the same. K(y) was chosen to have steep gradients, here defined as
C
K(y) B (A))(A) ((A-3) tanh(8r 5) (A- 15) tanh(8r + 5) + A ttan(A))),
(14)
where 9, B 0.0215, and C = 0.005. Once again, we learned the operator G cly, 0) c(y, 0.5). We again used the same RBF-FD solver to generate 1000 training and 200 test input/outpin function pairs (albeit at 4325 collocation points in 3D). We used eight spatial patches for the PoU trunks shown in Figure 6 The last column in Table shows that most of the ensemble DeepONets did poorly, as did the POD- DeD-DeepONet. However, the POD-PoU ensemble achieved almost a 2x reduction in error over the Danilla-DeepONet.
3.4 Runtime Comparison
The ensemble DeepONet architectures all have more trainable parameters than the vanilla and POD DeepOnets. This leads to higher training and inference times. We report the average time per training epoch and inference time on the test datasct in Tables 2 and respectively. The training times were larger in ensemble DecpONets with more trunk networks, considerably so when the PoU-MoE trunks were used (an order of magnitude increase in training time on the 3D reaction-diffusion problem). The inference times showed a similar trend, although much less pronounced (only half an order of magnitude slowdown the 3D problem). These slowdowns are because our current PoU-MoE implementation contains serial loop over the patches in the forward pass, leading to slower back-propagation over its parameters. Tn future work, we plan to address
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table Average time per training epoch in seconds. RD stands for reaction-diffusion
Darcy flow
Cavity flow 2D RD
3D RD
Vanilla
8.93e 4
3.99e
2.97e 4 2.10e 4
POD
5.19e 4
2.46e
2.06e 4 1.22e
Modified-PON
6.86e 4
2.49e
2.08e - 1.22e
(Vanilla, POD)
9.80e 4
3.92e
3.03e
2.32c
-4
(P+ 1a-Vanllla
1.10e 3
8.51e
7.27e 4 9.45e
4
Vanilla-PoU
8.67e 4
9.52c
1.03e 3
1.39c
POD-PoU
6.74e 4
8.21e
9.24e
4 1.28e
3
Vanilla-OOD-PoU
8,55e 4
9.48e
1.05e 3
1.43e
3
Table Inference time on the test dataset in seconds.
RD stands for reaction-diffusion.
Darcy flow
Cavity flow 2D RD
3D RD
Vanilla
1.66e 4
1.39e
1.32e
7.20e
POD
1,57E 4
1.12e
1.12e 4
6.42e
Modiffed-POD
1.34c 4
1.08c
9.94e 5
6.62c
5
(Vanilla, POD)
1.69e 4
1.33e
1.20e 4
7.76e 5
(P + 1)-
2.08e 4
2.12e
1.71e 4
1.48e
4
Vanilla-PoU
1.91e 4
2.42e
2.21e 4
2.37e 4
POD-PoU
1.63e 4
1.94e
1.96e
4
2.30e
Vanilla-POD-PoU
2.00c 4
2.18c
2.28e
2.41e
Table 4: Effectiveness of different trunk choices. The yes/no refers to whether the strategy beats a vanilla- DeepONet. The bolded results the best strategy for each experiment.
RD stands for reaction-difflsion..
Trunk Choices
Darcy flow
Cavity flow 2D RD
3D RD
Only POD global modes Only modified POD global modes Adding POD global modes Adding spatial locality
Yes
No
No
No
Yes
No
No
No
Yes
Yes
Yes
No
No
Yes
Yes
No
Only POD global modes + spatial locality
Yes
Yes
Yes
Yes
Only POD global modes spatial locality + mild vverparametrizationn
Yes
Yes
Yes
No
Adding excessive overparametrization
No
Yes
Yes
No
this with novel parallelization strategy; we believe this will speed up the ensemble archilectures with the PoU-MoE trunk considerably. It is also important to note that despite this increased cost, Table [ shows that the POD-PoU ensemble is more than 2x as accurate as the vanilla-eeopONct; the POD-DeepONet (and other ensembles) have errors that are two orders of magnitude worse!
4 Conclusions and Future Work
We presented the ensemble DeepONet, method of enriching DeepONet trunk with arbitrary trunks. We also developed the PoU-MoE trunk to aid in spatial locality, Our results demonstrated significant accuracy improvements over standalone DeepONets on several challenging operator lcarning problems, including a particularly challenging 3D problem in the unit ball. One of the goals of this work was to provide insight into choices for ensemble trunk members. Thus, we considered different combinations of three very specific choices: vanill-DeepONeet trunk (vanilla trunk), the POD trunk, and the new PoU-MoE, trunk. Our results (summarized in Table 4 make clear that while different ensemble strategies beat the vanila--eepONNet in different circumstances, only the POD-PoU ensemble consistently beats the vanilla-DeepONet across all
11

---

Published in Transactions on Machine Learning Research (03/2025)
problems. Simple overparametrization ((P -)-anlllla DeepONet) is not enough and sometimes deteriorates accuracy; a judicial combination of local and global basis functions is vital. Further, adding the PoU-MoF. trunk aids expressivity in every problem that involves steep spatial gradients in either the imput or output functions. Finally, it appears that the full benefits of the PoU-MoE trunk are mainly achieved when the POD trunk is also used in the ensemble.
Given the generality of our work, there are numerous possible extensions along the lines of problem-dependent choices for the ensemble members. The PoU-MoE trunk merits further investigation. It is plausible that adding adaptivity to the PoU weight functions could improve its accuracy further, as could spatially hierarchical formulation. Our work also paves the way for the use of other non-ncural network basis functions within the ensemble DeepONet.
References
Thomas Abeel, Thibault Helleputte, Yves Van de Poer, Pierre Dupont, and Yvan Sacys. Robust biomarker identification for cancer diagnosis with ensemble feature selection methods. Bioinformatics, 26(3):392 :392-398,
2010.
Kevin W Aiton and Tobin A Driscoll. An adaptive partition of unity method for chebyshev polynomial interpolation. STAM Journal on Scientifie Compuling, 40(1):A251-AD2 A265, 2018.
Wael Awada, Taghi M Khoshgoftaar, David Dittman, and Randall Wald. The effect of mumber of iterations on ensemble gene selection. In 2012 11th International Conference on Machine Learning and Applications, volume 2, pp. 198 203. IEEE, 2012.
M. Keith Ballard, Roman Amici, Varun Shankar, Lauren A. Ferguson, Michael Braginsky, and Robert M. Kirby. Towards an extrinsic, CG-XFEM approach based on hierarchical enrichments for modeling progressive fracture. Computer Methods in Applied Mechanics and Engincering. 388:114221, January 2022. ISSN 0045-7825. doi: 10.1016/j.cmma cmt..
Pau Batlle, Matthieu Darcy, Bamdad Hosseini, and Houman Owhadi. Kernel methods are competitive for operator learning. Journal of Computational Physics, 496:112549, January 2024. ISSN 0021-9991. doi:
10.1016/1.jp.2223..122 112.41.
Victor Bayona, Natasha Flyer, and Bengt Fornberg. On the role of polynomials in RBF-FD approximations: ITI. Behavior near domain boundaries. Journal of Computational Physics, 380:378-23 -998, 2019. doi:
10.0016/1.epp2018.1200...
T. Belytschko and T. Black. Elastic crack growth in finite elements with minimal remeshing. International Journal for Numerical Methods in Engineering, 45(5):601 620, 1999. ISSN 1097-0207. doi: 10.1002/(SICI)
Aniruddha Bora, Khemraj Shukla, Shixuan Zhang, Bryce Harrop, Ruby Leung, and George Em Karniadakis. Learning bias corrections for climate models using deep neural operators, February 2023. arXiv:2302.03173
[physics].
Zhiqiang Cai, Seokchan Kim, and Byeong- Chun Shin. Solution Methods for the Poisson Equation with Corner Singularities: Numerical Results. SIAM Journal on Scientific Computing, 23(2):672- 682, January 2001. ISSN 1064-8275. doi: 00.1377/116449855037777.. Mathematics.
Publisher: Society for Industrial and Applied
Roberto Cavoretto, Alessandra De Rossi, and Wolfgang Erb. Partition of Unity Methods for Signal Processing on Graphs. Journal of Fourier Analysis and Applications, 27(4))66, July 2021. ISSN 1531-5851, doi:
10.1007/:00041100--08711..
Nithin Chalapathi, Yiheng Du, and Aditi Krishnapriyan. Scaling informed hard constraints with mixture-of-experts,, 2024.
Anindya Chatterjee. An introduction to the proper orthogonal decomposition. Current Science, 78(7): 808 817, 2000. ISSN 0011 101999..1.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Newral Networks, 6(4):911- 917, July 1995. ISSN 1941-0093. doi: 10.1109/72.3992555 Conference Name: LEEE Transactions on Neural Networks.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards Understanding the Mixture-of- Experts Layer in Deep Learning. Advances in Neural Information Processing Systems, 35:23049 -2220, December 2022.
Hugh Chipman, Edward George, and Robert McCulloch. Bayesian ensemble learning. Advances in neural information processing systems, 19, 2006.
Ronan Collobert, Samy Bengio, and Yoshua Bengio. parallel mixture of syms for very large scale problems. Advances in Neural Information Processing Systems, 14, 2001.
B. Dasarathy and B. Sheela. A composite classifier system design: Concepts and methodology. Proceedings of the TEFE, 67 (5):7008 713, May 1979. ISSN 1558-2256. doi: 10.100//ROC..199.111 Conference Name: Proceedings of the IEEE.
David J Dittman, Taghi M Khoshgoftaar, Randall Wald, and Amri Napolitano. Comparing two new gene selection ensemble approaches with the commonly-used approach. In 2012 11th International Conference on Machine Learning and Applications, volume 2, pp. 184-191. 191. IEEE, 2012.
Xibin Dong, Zhiwen Yu, Wemming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Compuler Science, 14(2):241-2 -58, April 2020. ISSN 2095-2236. doi: 10.1007/911704-019888888
Haytham Elghazel and Alex Aussem. Unsupervised feature selection with ensemble learning. Machine Learning, 98:157-1800 2015.
Gregory E. Fasshaucr. Meshfree Approzimation Methods with MATLAB, volume 6 of Interdisciplinary Malhematical Sciences. World Scientific, 2007. ISBN 9789812706348.
Gregory E. Fasshauer and Michael J. McCourt. Kernel-based Approzimation Methods Using MATLAB, volume 19 of Interdisciplinary Mathematical Sciences. World Scientific, 2015. ISBN 978981488 3789814630139.
Natasha Flyer, Gregory A. Barnett, and Louis J. Wicker. Enhancing finite differences with radial basis Stokes equations. Journal of Computational Physics, 316:39-6) 62, July 2016. ISSN 0021-9991. doi: 10. 1...011//..o.2206.00...27
functions: Experiments on the Navier-Stokes
Charles Gadd, Sara Wade, and Alexis Boukouvalas. Enriched mixtures of generalised gaussian process experts. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Stalistics, volume 108 of Proceedings of Machine Learning Research, pp- 3144 3154. PMLR, 26 28 Aug 2020.
Donghai Guan, Weiwei Yuan, Young- Lec, Kamran Najecbullah, and Mostofa Kamal Rasel. A review of ensemble learning based feature selection. TETE Technical Review, 31 (3):1900 2014.
Jayesh K. Gupta and Jobannes Brandstetter. Towards Mutiisppaiioeepprral-saall Generalized PDE Modeling, November 2022. arXiv:2209.1561. [cs].
Ehsan Haghighat, Umair bin Wahcod, and George Karniadakis. En-DecpONet: An cnrichment approach for enhancing the expressivity of neural operators with applications to seismology. Computer Methods in Applied Mechanics and Engineering, 420:1 120:116681, February 2024. ISSN 0045-7825. doi: 10.1016/j.cma..0000 2023.116681. 166681.
Mingxuan Han, Varun Shankar, Jeff M. Phillips, and Chenglong Ye. Locally Adaptive and Differentiable Regression. Journal of Machine Learning for Modeling and Computing, 4(4), 2023. ISSN 2689-3967, 2689-3975. doi: 10. MarMachleerMMMoeCoooooo
Publisher: Begel House Inc.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Alfa Heryudono, Elisabeth Larsson, Alison Ramage, and Lina Sydow. Preconditioning for Radial Basis Function Partition of Unity Methods. Journal of Scientific Compuling, 67(3):1089-1109) June 2016. ISSN 1573-7691. doi: 10.1007//100155015--00
Amanda A. Howard, Sarah Н. Murphy, Shady E. Ahmed, and Panos Stinis. Stacked networks improve physics-informed training: applications to neural networks and deep operator networks, November 2023.
arXiv:2311.06483 (cs, math].
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79 87, 1991. doi: 10.1162/neo..991.33
Pengzhan Jin, Shuai Meng, and Lu Lu. MIONet: Learning Multiple-Input Operators via Tensor Product. SIAM Journal on Scientific Computing, 44(6):A3490-AM A3314,, December 2022. ISSN 1064-8275. doi: 10.1177/221147755.. Publisher: Society for Industrial and Applied Mathematics.
M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the em algorithm. In Proceedings of 1993 International Conference Newral Networks LLJCNN-99-NNap Japan), volume 2, pp. 1339-1344 vol.2, 1993. doi: 10.1109/IJCNN 199....999.... 1993.716791.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization, January 2017. arXiv:1412.6980 [cs].
Anders Krogh and Peter Sollich. Statistical mechanics of ensemble learning. Physical Review E, 55)1):811, 1997.
Elisabeth Larsson, Victor Shcherbakov, and Alfa Heryudono. A Least Squares Radial Basis Function Partition of Unity Method for Solving PDEs. SIAM Journal on Scientific Computing, (966):A2538A225)) Jamuary 2017. ISSN 1064-8275. doi: 10.1 10.113/11M1111800 Mathematics.
Publisher: Society for Industrial and Applied
Sanghyun Lee and Yconjong Shin. On the training and generalization of deep operator networks, September 2023. arXiv:2309.01020 [cs, math, stat].
Yun Li, Suyan Gao, and Songcan Chen. Ensemble feature weighting based on local learning and diversity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pp. 1019-1025, 2012.
Zongyi Li, Nikola Kovachki, Kamyar Aziwzadenehheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. In Proceedings of the 94th International Conference on Newral Information Processing Systems, NeurIPS '20, pp. 6755 6766, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-954-.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima, Anandkumar. Fourier neural operator for parametric partial differential equations, 2021.
Clodoaldo AM Lima, André LV Coelho, and Fernando J Von Zuben. Hybridizing mixtures of experts with support vector machines: Investigation into nonlinear dynamic systems identification. Information Sciences, 177(10))2049-2) 2044-204,, 2007.
Clodoaldo AM Lima, André LV Coelho, and Fernando J Von Zuben. Pattern classification with mixtures of weighted least-squares support vector machine experts. Neural Computing and Applications, 18:843-860, 2009.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via. DeepONet based on the universal approximation thcorem of operators. Nature Machine Intelligence, 3(3):218-229) March 2021. ISSN 2522-5839. doi: 10.1038/s42256-0222 022-2222550021-000------- Publisher: Nature Publishing Group.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data. Computer Methods in Applied Mechanics and Engineering. 393:114778, April 2022. ISSN 0045-7825. doi: 10.1016/j.cma.20 20..mm.......217777
Saced Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey. Artificial Intelligence Review, 42(2):275 293, August 2014. ISSN 1573-7462. doi: 10.1007 1..1007/110462-012-93333
Jeffrey S. McQuien, Kevin H. Hoos, Lauren A. Forguson, Endel V. Iarve, and David H. Mollenhauer. Geometrically nonlinear regularized extended finite element. analysis of compression after impact in composite laminates. Composites Part A: Applied Science and Manufacturing, 134:105907, July 2020. ISSN 1359-835X. doi:
J. M. Melenk and Babuvska. The partition of unity finite element method: Basic theory and applications. Computer Methods in Applied Mechanics and Engineering, 139(1))289-38)) 334, December 1996. ISSN 0045-7825. doi: 00006//00055-725(00000
Jun Wei Ng and Mare Peter Deisenroth. Hierarchical mixture-of-experts model for large-scale gaussian process regression. Xixi preprint 1xX.v:1412.9073 2014.
Vivek Oommen, Khemraj Shukla, Saaketh Desai, Remi Dingreville, and George Em Karniadakis. Rethinking materials simulations: Blending direct numerical simulations with neural operators, December 2023. arXiv:2312.0544 [physics].
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Avizzadeneshelii Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators, February 2022. arXiv:2202.1121 [physics].
Ahmad Peyvan, Vivek Oommen, Ameya D. Jagtap, and George Em Karniadakis. RiemannONets: Interpretable neural operators for Riemann problems. Compuler Methods in Applied Mechanics and Engineering, 426:116996, June 2024. ISSN 0045-7825. doi: 10.1016/j.cma, 202.11911.66996
Yongiun Piao, Minghao Piao, Kiejung Park, and Keun Ho Ryu. An ensemble correlation based gene selection algorithm for cancer classification with gene expression data. Bioinformatics, (3324))3300-2) 2012.
Robi Polikar. Ensemble Learning. In Cha Zhang and Yunqian Ma (eds.), Ensemble Machine Learning: Methods and Applications, pp. 1-34. Springer, New York, NY, 2012. ISBN 978-1-4419-9826--. doi:
1900077978-1-44999 99119-93267_1..
Bogdan Raonic, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Convolutional neural operators. In ICLR 2023 Workshop on Physics for Machine Learning, 2023.
Yvan Sacys, Thomas Abeel, and Yves Van de Peer Robust feature selection using ensemble feature selection techniques. In Machine Learning and Knowledge Discovery in Dalabases: Europeun Conference, ECML PKDD 2008, Antwerp, Belgium, September 15-19, 2008, Proceedings, Part 11 19, pp. 313 325. Springer, 2008.
Ali Rafdari-Vaighann Alfa Heryudono, and Elisabeth Larsson. A Radial Basis Function Partition of Unity Collocation Method for Convection Diffusion Equations Arising in Financial Applications. Journal of Scientific Computing, 64(2):341-388 337, August 2015. ISSN 1573-7691. doi: 10.1007/s10915--0449933
Varun Shankar and Aaron L. Fogelson. Hyperviscosity based stabilization for radial basis conction-finitt difference (RBF-FD) discretizations of advection diffusson equations. Journal of Computational Physics, 372:616 639, November 2018. ISSN 0021-9991. doi: /tw.11/6...p..2211.0.....
Varun Shankar and Grady B. Wright. Mesh-free somi-Lagrangian methods for transport on a sphere using radial basis functions. Journal of Computational Physics, 366:170-190, August 2018. ISSN 0021-9991. doi: 10. 10.1016//.jcp..011 000.0....p..0.....0000
15

---

Published in Transactions on Machine Learning Research (03/2025)
Varun Shankar, Grady B. Wright, and Aaron L. Fogelson. An efficient high-order meshless method for advection-diffusion equations on time-varying irregular domains. Journal of Compulational Physics, 445: 110633, November 2021. ISSN 0021-9991, doi: 00.0016/...g22221111033...
Victor Sheherbakov and Flisabeth Larsson. Radial basis function partition of unity methods for pricing vanilla. basket options. Computers 8 Mathematics with Applications, (1()1185,,000 January 2016. ISSN 0898-1221. doi: (0006//.camwa.225.11...000
Khemraj Shukla, Vivek Oommen, Ahmad Peyvan, Michael Penwarden, Nicholas Plewacki, Luis Bravo, Anindya Ghoshal, Robert M. Kirby, and George Em Karniadakis. Deep neural operators as accurate surrogates for shape optimization. Engineering Applications of Artificial Intelligence, 129:107615, March 2024. ISSN 1952-1976. doi:
20w.aaapppa.ii......... 202310111..
Nathaniel Trask, Amelia Henriksen, Carianne Martinez, and Eric Cyr. Hicrarchical partition of unity networks: fast multilevel training. In Bin Dong, Qianxiao Li, Lei Wang, and Zhi-Qin John Xu (eds.), Proceedings of Mathematical and Scientifie Machine Learning, volume 190 of Proceedings of Machine Learning Research, pp. 271 286. PMLR, August 2022.
Eugene Tuv. Ensemble learning. Feature extraction: foundations and applications, pp. 187- 187-204, 2006.
Sofie Van Landeghem, Thomas Abeel, Yvan Saeys, and Yves Van de Peer. Discriminative and informative features for biomolecular text mining with ensemble feature selection. Bioinformatics, 26(18):1554 i560, 2010.
Holger Wendland. Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree. Advances in Computational Mathematics, 4(1):389-333 -96, December 1995. ISSN 1572-9044. doi: 10.1007 ALL007/BF0223445
Holger Wendland. Scaltered Dala Approsimation. Cambridge University Press, 2005. ISBN 9780521843355.
Chao Yuan and Claus Neubauer. Variational mixture of gaussian process experts. Advances in neural information processing systems, 21, 2008
Seniha Esen Yuksel, Joseph N. Wilson, and Paul D. Gader. Twenty Years of Mixture of Experts. IEFE Transactions on Neural Networks and Learning Systems, 23(8):1177-1191 August 2012. ISSN 2162-2388. doi: 10.1109/TNNLS TtTp//NNNS...1.22222222222 2012.2200022 Conference Name: IEEE Transactions on Neural Networks and Learning Systems.
Zecheng Zhang, Leung Wing Tat, and Hayden Schaefler. BelNet: basis enhanced learning, mesh-free neural operator. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 479(2276): 20230043, August 2023. doi: 10.1098 10.1098/rspa 202....02.004...
Publisher: Royal Socicty.
Zhi-Hua Zhou. Ensemble Learning. In Zhi-Hua Zhou (ed.), Machine Learning, pp. 181-210. Springer, Singapore, 2021. ISBN 9789811519673.
doi: 10.1007/978-9811 15-1967-3 8.
16

---

Published in Transactions on Machine Learning Research (03/2025)
A Ensemble Learning and mixture-of-Experee (MoE)
The key idea behind ensemble learning is to combine diverse set of learnable features from individual models
into a single model (Polikar 2012 Dong et al. 2020
Zhou
2021
Dasarathy & Shoola 1979). This technique
has been used for both supervised and unsupervised feature selection in a variety of applications (Sayys et al,
2008 Li et al. 2012 Eighazel & Aussem 2015
Abeel et al.
2010: Van Landeghem et al., 2010 Awada et al,
2012
Dittman et a.. 2012 Piao et al. 2012 Guan et al.
(2014) draws an important distinction between
using ensemble learning for feature selection and using feature selection for ensemble learning (where the
former category is known to overcome the problem of local minima in machine learning). Tt is generally
known that these methods are more stable than single base learners (Tuy 2006 Guan et al., 2014). While ensemble methods have Iraditionally been studied with statistical perspective (Krogh & Sollich 1997 Chipman et al. 2006), we focus more on its feature selection capability, i.e. our work falls in the category of ensemble lcarning for feature selection. We use ensemble learning specifically to aggregate the global and local spatial features learned by the vanilla, POD, and PoU-MoE trunks into a single ensemble trunk.
MoE, first introduced in (Jacobs al. 1991 Jordan & Jacobs 1993); a method which an "expert" model
focuses on learning from subset of the training dataset. These models can be support vector machines (Lina
et al.
2007
Lima et al. 2009 Collobert et. al. 2001), Gaussian Processes (Ng & Deisenroth 2014: Yuan &
Neubauer
2008 Gadd et al. 2020), and neural networks. Similar to ensemble learning, the MoE idea has
also
proven to be very succossful in diverse MI. applications (Masoudnia & Ebrahimpour 2014 Yuksel et al.
2012
Chen et al. 2022). Most recently, MoE has also been used in physics-informee learning; Chalapathi
et al. (2024) uses MoE across the spatial domain with non-overlapping patches to decompose global physical hard constraints into multiple local constraints, Our PoU-MoE trunk uses a similar methodology where it has individual expert trunk networks on each patch in the domain, albeit with overlapping patches. However, instead of learning physical constraints, we let our experts learn spalially local basis functions (see Section 3.2 for further discussion this spatial locality).
B Speculation on an ensemble FNO
Here, we show one possible technique for incorporating the PoU-MoE localized bascs into the FNO architecture, i.e., we show how to create an ensemble FNO. FNOs consist of a lifting operator that lifts the input functions to multiple channels, projection operator that undoes the lift, and intermediate layers (Fourier layers) consisting of kernel-based integral operators discretized by the fast Fourier transform (FFT); these integral operators are also typically augmented by pointwise convolution operations. Let fe denote the intermediate function at the th Fourier layer. Then, the output Je+1 of this layer (and the input to the next layer) is given by
())(() (f do + W.ll)) x E S
fe+1()) = 1o
(15)
where a an activation function applied pointwise, K. a matrix-valued kernel learned in Fourier space via the FFT, and W is the aforementioned pointwise convolution (Li et al. (2021). Since FNOs use the FFT to compute the integral operator in (15), this effectively constitutes projection of fi(x:) onto set of globol Fourier modes (trigonometric polynomials complex exponentials). One possible method for creating ensemble FNO would involve modifying (5) to incorporate set of localized basis functions using the PoU-MoE formulation as follows:
/ K(xx K(a,)))((()) dr +
P Σ wn(U) I K(x,v) fe())) dr + W Sn.
f4411)= a
(16)
Global basis
Localized basis
where P is the number of spatial patches (all of which are hypercubes). The PoU-MoE formulation now combines a set of localized integrals on cach patch, each of which when computed by an FFT would constitute a projection of f. (restricted to Q2g) onto local Fourier basis. This loosely resembles the Chebyshev polynomial PoU approximation introduced by Aiton & Driscoll (2018).
17

---

Published in Transactions on Machine Learning Research (03/2025)
It is worth mentioning that this is one of many ways to combine different basis functions in FNOs. Another way is to introduce set of local basis functions at the final projection operator that maps to the output function. The projection operator's final layer can be enlarged to weight the additional basis functions, closely resembling how the branch weights the ensemble trunk in ensemble DeepONets. Similar extensions are possible for the GNO and even Bernel/GP-baser operator learning techniques.
C Suboptimal ensemble trunk architectures
We document here our experience with other ensemble trunk architectures. We primarily made the following two other altempts:
A residual ensemble: Our first attempt was to combine the different trunk outputs using weighted residual connections with trainable weights, then activale the resulting output, then pass that activated output to a dense layer. For instance, given two trunks Tı and T2, this residual ensemble trunk would be given by
f frs Wo (tanh(w))7 + (anh(w))72) b,
(17)
where a was some nonlinear activation, W was some matrix of weights, and b bias. We also attempted using the sigmoid instead of the lanh. The major drawback of this architecture was that the output dimensions of the individual trunks had to match, te., P1 = to add the results (otherwise, some form of padding would be needed). We found that this architecture indced outperformed the Vanila- in some of our test cases, but required greater fine tuming of the output dimension p. In addition, found that this residual ensemble failed to match the accuracy of our final ensemble architect
An activated ensemble: Our second attempt resembled our final architecture, but had extra activation function and weights and biases. This activated ensemble trunk would be given by
fact Wo ([TI, +
(18)
This architecture allowed for different p dimensions (columns) in and T2- However, found that this architecture did not perform well when the POD trunk one of the constituents of the ensemble; this is likely because it is suboptimal to activate a POD trunk, which is already data-dependent basis. There would also be no point in moving the activation function onto the other ensemble trunk constituents, since these are always activated if they are not POD trunks. Finally, though and allowed for a trainable combination rather than simple stacking, they did not offer greater expressivity over simply allowing wider branch to combine these different trunks. We found that this architecture also underperformed our final reported architecture.
18

---

Published in Transactions on Machine Learning Research (03/2025)
D
Proof of Universal Approximation Theorem for the PoU-MoE DeepONet
We have
P wy (v)
(g)()(() (u)(())
(())(()
((u 0b). Σ
b0
P
P
Σ my(())
g(u)(v) - B(u:
Σ
(w)()),((v0+t))
wy
P
Σ
ws((v)
bo
Σ my(y) (G(w)(v) (o), 0+,)) bbo
1v
wy (v(W) - (B(u; (,,; (,,))
-bollv.
Given branch network & that approximate functionals to arbitrary accuracy, the (generalized) universal approximation theorem for operators automatically implies that (Chen & Chen 1995 Lu et al. 2021 a trunk network T, (given sufficient capacity and proper training) can approximate the restriction of to the support of wy(y) such that: (v) - (B(u; 0,), (v:00,,)) - -boll S Ej:
for all y in the support of wy and any Eg > 0. Setting ej = e, j= I....,P. P. we obtain:
P
[((()(v) - (f))w)))((w)))ll Se Σ wc)))
(I(())()) (T) Sc.
where 6 > can be made arbitrarily small. This completes the proof.
E Hyperparameters
E.1 Network architecture
In this section, we describe the architecture details of branch and trunk networks. The architecture type, size, and activation functions are listed in Table 5 The CNN architecture consists of two fivo-filter convolutional layers with 64 and 128 channels respectively, followed by a lincar layer with 128 nodes. Following Lu et al. (2021), the last layer the branch network does not use an activation function, while the last layer in the trunk does. The individual PoU-MoE trunks in the ensemble models also use the same architecture as the vanilla trunk. We use the unstacked DeepONet with bias everywhere (except the POD-DeepONet which does not use a bias).
E.2 Output dimension p
We list the relevant DeepONet hyperparameters we use below. The p (Pron for POD) values are listed in Table 0 for all the DeepONets.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Table DeepONet network architectures across all models and problems. The CNN architecture is described in Appendix
E.
Branch
Trunk
Activation function
Darcy flow Reaction-Diffusinn
3 layers, 128 nodes
3 layers, 64 nodes
Leaky-ReLU
2D
CNN
3 layers, 128 nodes
ReLU tanh
Cavity flow
CNN
[128, 128, 128, 100]
3D Reaction-Diffusion
3 layers, 128 nodes
3 layers, 128 nodes
ReLUU
Table 6: p (Pron for POD) values for the various DeepONet models. For (P va-anilaa DeepONet, the total number of basis functions is shown below. RD stands for reaction-diflusion. Darcy flow Cavity flow
2D RD 3D RD
Vanilla POD
100
100
100
100
20
6
20
20
Modified-PON
20
6
20
20
(Vanilla, POD)
(100, 20)
(100,
(100,20)
(100,20)
(P + Vanilla
400
500
700
900
Vanilla-PoU
(100, 100)
(100,100)
(100, 100)
(100,100)
POD-PoU
(20, 100)
(6, 100)
(20, 100)
(20, 100)
VanillaPOD-PPoU
(100, 20, 100)
(100, 6, 100)
(100, 20, 100)
(100, 20, 100)
E.3 Partitioning
The PoU-MoE trunk has certain hyperparameters that must be chosen. In our experiments, to maximize accuracy, chose the patch size and the number of patches that produced the smallest possible patches and the smallest number of patches, while simultaneously seeking that the domain was covered and ensuring that the patches did not extend too far outside the domain boundary. Coincidentally, this strategy coincided with placing individual patches over regions of high spatial error in the vanilla-DeepONet solution (effectively, patches over 'features of interest'). In the reaction-diffusion examples, even though we used uniform patch radii, we ensured that the patches did not overlap horizontally over line of the discontinuily. This choice combined with the use of ReLU activation ensured that we resolved that discontiniity better than vanilla- DeepONet; believe this is one of the unique strengths of the PoU-MoE approach. Currently, we use the same trunk architectures on each patch as in the vanilla-DeepONes In future work, adaptive patch selection strategies (such as making the patch centers and radii trainable enforcing soft constraints them as part of the loss function) can be used to automate determining patch placeeent and patch size. Furthermore, the patch shape can be changed depending on the problem domain; clongated mangated/ellipsidda patches can be used in narrower regions where spherical patches are not well suited.
F Additional Results
We present additional results and figures in this section related to the problems in Section
3
F.1
2D
Darcy flow
The 2D Darcy flow problem models fluid flow within porous media. The flow's pressure field u(y) and the boundary condition are given by
-V - (K(() Vu(y)) = f()), y E Ω u(y) ~ gP (0, (()h())))))
(19)
(20)
where (yy) the permeability field, and f(y) is the forcing term. The Dirichlet boundary condition was sampled from a zero-mean Gaussian process with a Gaussian kernel as the covariance function; the kernel length scale was o 0.2. As in Lu al. (2022). we learned the operator u(y)lon + u(y)la- We used the dataset provided in Lu et al. (0222) which contains 1900 training and 100 test input and output function
20

---

Published in Transactions on Machine Learning Research (03/2025)
A
B
C
Input function
Output function
PoU partitioning
Vanilla
(Vanilla, POD, PoU)
(POD, PoU)
24
10
y1
J1
Figure 7: The 2D Darcy flow problem. (A) and (B) show example input and output functions respectively. (C) shows the three patches used for the PoU-MoE trunk. (D), (E), and (F) show the spatial mean squared error (MSE) for the vanilla, ensemble vanill-POOD-POU, and ensemble POD-PoU DeepONets respectively.
pairs. Ω was triangular domain (shown in Figure 7) The permeability field and the forcing teri were set to (y) = 0.1 and f(v) = -. Example imput and output functions, and the three patches for PoU trunks are shown in Figure 7 The partitioning always ensures that the regions with high spatial gradients are captured completely or near-completely by patch.
We report the relative l errors (as percentages) on the test dataset for the all the models in Table I The Pon- ensemble was the most accurate model with 4.5x error reduction over the vanilla- DeepONet and 1.5x reduction 3.7x error reduction over the vanilla-DeepONeer and 1.5x reduction over the POD-DeepONet. The highly overparametrized (P- 1)-vanilla model was less accurate than the standalone DeepONets. On this problem, overparamctrization appeared to help only when spatial localization was also present; the biggest impact appeared to be from having both the right global and local information. The MSE errors as shown in Figure corroborate these findings.
our POD-DeepOnet The POD-PoU ensemble was second best with
21


Reviews:
[{"review_id": "U4ZXtQU8I1", "reviewer": "Reviewer_a3Dm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Nice numerical examples are comprehensive, covering several PDEs and dimensions.\n* The methodology is well-structured and logically presented.\n\n\nWeakness:\n* Overall clarity, including notation, references, and figure captions, needs improvement. See minor issues.\n* Theorems 1 and 2 are misleading in their current form.\n* The description of the POD-PoU method is unclear, and its relationship to the POD-DeepONet is not well-explained.\n* Certain numerical results (e.g., Table 1 in the 3D RD case) are confusing and insufficiently justified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an ensemble DeepONet framework for PDE operator learning. The main contribution of this work is introducing a partition of unity approach to localize the trunk-net basis, while simultaneously incorporating global POD trunk bases. Through several numerical examples, the proposed method demonstrates significantly improved accuracy, outperforming other ensemble DeepONet approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Main issues:\n\nThe statements in Theorem 1 and 2 are misleading. In the current statement, it is suggested that the size of the DeepONet, and the trained parameters are independent of \\epsilon. In the original universal approximation theorem in [Lu et al, 2021], it is clearly that these parameters all depend on \\epsilon. \n\nThe description of POD-PoU method in section 2.2.2 is not clear. Most of the current section 2.2.2 is describing the POD-DeepONet instead of POD-PoU. Is the idea of POD-PoU simply doing an SVD to the learned PoU-MoE basis \\tau_k in (3), or simply stacking the POD basis of v_i’s to the PoU basis?\n\nThe clarity of the numerical section needs to be improved. See more details in minor issues.\n\nTable 1, 3D RD: a weird observation here is large deviation (larger than the average!) for all methods except vanilla DeepONet, modified-POD, and POD-PoU. This somehow does not happen to the 2D case at all. What is more confusing is that Vanilla,POD,PoU is significantly worse than POD,PoU.\nCould you briefly explain this phenomenon? \n\nMinor issues:\n\nEqn (2), the variable u in \\hat{\\mathcal{G}} should be u(X), and needs to be distinguish from u in \\mathcal{G}(u).\n\nEqn (4), is y^c supposed to be y^c_k? Also the phi_k function notation is abused in eqn (4) and (5). It is defined as a bi-variate function on R^d \\times R^d but used as a uni-variate in (4) and (5).\n\nSection 2.2.2, I think the reference (Chatterjee, 2000) to POD-DeepONet is wrong.\n\nSection 2.2.2, should T = \\frac{1}{N} V^\\top V? The eigenvectors of \\frac{1}{N}  VV^\\top is N dimensional but the basis should be N_y dimensional.\n\nFigure 2c, the sentence “… has learned a global basis function (right)” is confusing; the figure itself seems to be a local function.\n\nFigure 3 DEF, are you plotting error for the u-component or v-component? This should be specified in the caption.\n\nSection 3.3 should be a subsubsection.\n\nSection 3.5, the last sentence is a bit confusing; Are the authors suggesting ensembles with PoU-MoE trunk are slower but more accurate? This is not true for vanilla,PoU and Vanilla,POD,PoU\n\n\n\n\nTypos:\n\nThe norm notation is eqn (2) is not clear. Should it be a supremum over all $y$ and all $u$? Same for (6).\n\nPage 6, do you need to sum over different functions in e_{mse}?\n\nFigure 3A, the ylable should be u_b(x)?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DMoItGJZbM", "reviewer": "Reviewer_M2jA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a novel approach to operator learning using ensemble and Mixture-of-Experts (MoE) DeepONets. The authors demonstrate significant improvements in accuracy and expressivity, particularly when combining global and local basis functions. Below are the key strengths and weaknesses of the submission:\n\n> **Strengths**\n\n- **Novel Architectures**: The ensemble DeepONet and PoU-MoE DeepONet architectures are innovative, providing a framework for incorporating basis enrichment and spatial locality into operator learning. [Refer to the weaknesses section for related concerns.]\n- **Theoretical Foundation**: The paper establishes a robust theoretical foundation by proving the universal approximation capability of both the ensemble DeepONet and the PoU-MoE DeepONet. This demonstrates that the proposed architectures are theoretically sound and capable of approximating a broad range of operators.\n- **Results**: Experimental results highlight the effectiveness of the proposed architectures across various challenging operator learning problems, particularly those with sharp gradients. The reported 2-4x reduction in relative ℓ2 errors compared to standard DeepONets and POD-DeepONets underscores the significant accuracy improvements achieved.\n- **Insightful Analysis**: The authors provide a thorough analysis of the performance of different ensemble DeepONet configurations. They demonstrate that simple over-parameterization is insufficient and that a judicious combination of local and global basis functions is critical for achieving state-of-the-art accuracy. This analysis deepens the understanding of the proposed approach and its effectiveness.\n\n\n> **Weaknesses**\n\n1. **Generalization**: This is arguably the most critical weakness of the paper. The results do not convincingly demonstrate the generalizability of the proposed method, which is essential for establishing a framework for operator learning. Specifically:\n   - The results in Table 1 do not adequately prove the method's generalizability. While the approach works on different types of problems, the evidence suggests potential overfitting and reliance on specially devised inductive biases for specific cases (e.g., initial and boundary conditions) of Darcy, cavity flow, and reaction-diffusion problems.\n   - A truly general operator learning model should be able to handle varying initial and boundary conditions within the same parent set of partial differential equations (PDEs). The use of basis functions like POD tailored for particular flow conditions introduces data leakage. The authors should address this issue, as discussed further in the \"Requested Changes\" section.\n\n2. **Limited Interpretability of Assumptions and Results**: The interpretability of both the results and the impact of the underlying assumptions is limited. For example:\n   - The effect of choosing the PoU partition, as shown in Figures 3 and 4, is unclear. How does this choice influence the results? Could alternative partitions yield better outcomes?\n   - Why is the PoE trunk model expected to perform well in this context? The reasoning behind these choices should be explicitly clarified.\n\n3. **Comparison with Previous Work**: The paper does not sufficiently clarify how it builds upon or diverges from prior work in the field. While related work is cited in Section 1.1, the discussion lacks explicit statements about how the current approach differs from or advances existing methods.\n\n4. **Clarity in Novelty**: As noted in the authors’ Proof of Theorem 1, the ensemble DeepONet does not require logic beyond the original DeepONet formulation (Lu et al.). This raises questions about whether the proposed method represents a novel contribution to operator learning or merely an application of an existing method.\n\n5. **Nomenclature**: The terminology used could be more precise. For instance, the inner product over the trunk components in Eq. (1) and Section 2.2 is referred to as \"Ensemble Mixture of Experts (MoE).\" However, in machine learning, ensemble models are typically independent and map input to output variables independently. The authors should reconsider or clarify this nomenclature.\n\n\n> **Conclusion**\n\nAddressing these weaknesses would significantly enhance the paper’s clarity, comprehensiveness, and impact. Furthermore, the authors’ proposed future work on parallelisation strategies and adaptive patch selection is promising and could substantially improve the practicality of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents two novel DeepONet (Deep Operator Network) architectures for operator learning: the **ensemble DeepONet** and the **spatial mixture-of-experts (MoE) DeepONet**. The authors demonstrate the effectiveness of both approaches, particularly when combined, in improving expressivity and generalisation capabilities.\n\n> **Ensemble DeepONet:**\n\n-  This architecture allows the enrichment of a DeepONet's trunk network with multiple distinct trunk networks, facilitating greater expressivity and accuracy.\n-   The authors prove that the ensemble DeepONet is a **universal approximator**.\n-  Specifically, this means an ensemble DeepONet $\\hat{G}$ can approximate a continuous operator $G$ to any desired accuracy:\n    $$\n    \\| G(u)(y) - \\hat{G}(u)(y)\\|_V \\le \\epsilon\n    $$\n    where $\\epsilon > 0$ can be made arbitrarily small.\n\n> **Spatial Mixture-of-Experts (MoE) DeepONet:**\n\n*   This architecture introduces a **partition-of-unity (PoU) MoE trunk network** that promotes spatial locality and model sparsity.\n*   The PoU-MoE trunk utilizes spatially-localized, overlapping, distinct trunks blended smoothly using compactly-supported weight functions.\n*   The authors prove that the PoU-MoE DeepONet is also a **universal approximator**.\n\n> **Experimental Results:**\n\n*   The authors compare various ensemble DeepONet formulations (including vanilla, POD, and PoU-MoE trunks) against vanilla and POD DeepONets.\n*   They demonstrate that **ensemble DeepONets with a combination of global (POD) and local (PoU-MoE) trunks achieve the best performance** across a range of challenging 2D and 3D operator learning problems, including:\n    *   Lid-driven cavity flow\n    *   Reaction-diffusion problems with sharp gradients\n    *   Darcy flow\n*   These ensembles achieve **2-4x lower relative ℓ2 errors** than standard DeepONets and POD-DeepONets.\n*   The authors show that **simple over-parametrization without spatial locality is insufficient** for achieving state-of-the-art accuracy.\n\n> **Limitations:**\n\n*   Ensemble DeepONets with PoU-MoE trunks require more training time due to a larger number of parameters. However, the authors plan to address this with parallelization strategies in future work.\n*   The experiments were conducted with an unstacked branch network, which may have resulted in slightly lower accuracy compared to a stacked branch approach.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the strengths and weaknesses identified in our previous conversation, here are some proposed adjustments to the submission, categorized by their importance. Currently, I have marked \"No\" for claims and evidence, as I recommend the authors revise their script to address the weaknesses I have highlighted.\n\n> **Critical Adjustments:**\n\n- **Dataset**: Given that the problem is an unsupervised learning problem, the authors should provide a detailed explanation of how the data flows through the problem, ideally with a figure illustrating the overall method. This should include how data is processed and which parts are responsible for learning. At present, Figure 1 is not sufficiently informative.\n\n- **Generalization of the Method Across Different Initial and Boundary Conditions**: To address generalization correctly, the authors should detail a comparison where the model is trained on one boundary condition and evaluated on its ability to generalize to other initial and boundary conditions. This would demonstrate the model's capability as a neural operator. Even if the results are not optimal, they would still provide valuable insights.\n\n- **Interpretability of Results**: The authors should offer insight into their solution and comparisons among different models. In the results section, please clarify your understanding of the solution and explain how the inductive bias contributes to the results obtained.\n\n- **Related Work and Comparison**: The introduction and related work sections could benefit from including other studies that use neural networks as operators. I conducted a preliminary search and found works that may be relevant, such as K. Michałowska, S. Goswami, G. E. Karniadakis, and S. Riemer-Sørensen's , \"Neural Operator Learning for Long-Time Integration in Dynamical Systems\", \"Convolutional Neural Operators for robust ...\" (https://arxiv.org/abs/2302.01178), \"Neural Networks Can Be FLOP-Efficient Integrators ...\" (https://arxiv.org/abs/2404.05938), Graph neural network operators: a review (https://link.springer.com/article/10.1007/s11042-023-16440-4) etc. I searched using terms like \"Mathematical function and operator learning with Neural network.\" While these references may help, I recommend conducting a search for other recent and relevant studies yourself.\n\n- **[Optional] Comparative Experiments with Stacked and Unstacked Branch Networks**: The authors should perform comparative experiments using both stacked and unstacked branch networks. This could provide valuable insights into the accuracy gains achieved with a stacked branch network and help assess the trade-offs between complexity and performance. Although the authors mention this as a limitation, the current sources do not provide information about the accuracy improvements associated with using stacked branch networks.\n\n> **[Optional] Adjustments with the PoU Method:**\n\n- **Exploration of Patch Selection Strategies**: The authors should explore alternative patch selection strategies for the PoU-MoE trunk and evaluate their impact on performance. This could include:\n  - **Adaptive Patch Selection**: Investigate making the patch centers and radii trainable or applying soft constraints on them as part of the loss function. This would allow the network to learn the optimal patch configuration for each problem automatically. The authors note adaptive patch selection as a potential future direction.\n  - **Problem-Specific Patching**: Explore using different patch shapes and sizes tailored to the problem domain, such as elongated or ellipsoidal patches in regions with anisotropic features. This would emphasize the importance of adequately covering regions with high spatial gradients.\n\nAddressing these critical adjustments would significantly enhance the completeness of the paper and strengthen its case moving forward. Implementing the optional adjustments would further improve the paper's clarity, comprehensiveness, and impact, making it a more valuable contribution to the field of operator learning.", "broader_impact_concerns": "While the sources do not explicitly detail any ethical concerns, they do mention limitations regarding the computational cost of the proposed ensemble DeepONets, especially when using PoU-MoE trunks. This raises a potential concern about the accessibility of the technology, which could be discussed in a broader impact statement.  \n\n*   **Environmental Impact of Computation:** The sources state that ensemble DeepONets require more time to train, especially those using PoU-MoE trunks. The sources also acknowledge the need for future work to ameliorate this issue with parallelization.  However, the sources do not quantify how much longer these models take to train or the energy consumption of such training.   Training large models can have a significant environmental impact due to the energy required for computation. A broader impact statement could address this by discussing:\n    *   Strategies to improve energy efficiency during training, such as using more efficient hardware or algorithms.\n    *   The trade-off between model accuracy and computational cost, and the implications for the environmental footprint.\n\n*   **Accessibility of Advanced Techniques**: The increased computational cost of ensemble DeepONets, especially those incorporating PoU-MoE trunks, could potentially limit their accessibility to researchers and practitioners with limited computational resources.  This could create a disparity in the ability to benefit from these advancements, potentially exacerbating existing inequalities in research and development. A broader impact statement could address this concern by:\n    *   Discussing efforts to make the technology more accessible, such as developing more efficient implementations or exploring techniques to reduce the computational burden without sacrificing accuracy.\n    *   Highlighting the importance of considering the computational cost when designing and deploying machine learning models, promoting responsible innovation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rYZAiCc0eo", "reviewer": "Reviewer_qafA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem under study is highly relevant in practice, and the relevant literature seems adequately surveyed. The paper conducts detailed experiments with well-documentation and sound design. The paper itself is easy to follow. The ideas introduced are simple and intuitive.\n\nSince there are more parameters in the proposed architecture, a more detailed study on the trade-off between accuracy vs. computational and data efficiency is needed to make the argument more convincing. And related to this, how much effort is required in tuning the additional parameters introduced? I have some doubts on the performance of the algorithm without some careful choices of the hyper-parameters and/or prior knowledge of the problem. So some principled justification on how to choose these in practice is desirable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose ensemble DeepONet where multiple learnable trunk networks are stacked together with a single branch network via inner product to produce the prediction. Universal approximation results are given for the resulting operator. \n\nSome choices for the ensemble trunk members under consideration are: (1) a PoU-MoE trunk is proposed where several trunk nets are trained using data from each overlapping patch and linearly combined together into a single final trunk; (2) POD trunk, which is similar to a eigen-decomposition and keeping the top eigenvectors as basis (i.e., spectral decoder), which means that the POD trunk has no trainable parameters.\n\nExperiments are conducted using various combinations of ensemble choices listed above, and it is observed that blending global and local basis functions improve accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It seems to me that the idea of PoU-MoE trunk goes very much in the direction of multi-resolution analysis in e.g., signal processing (and there are neural operators built using these frame-like ideas). But unlike these, the choice of local radius $\\rho_k$ here would require prior knowledge of the problem and therefore not automatically adaptive. On the other hand, if one does have knowledge on the problem to guide the choice of $\\rho_k$, isn't a GNN architecture based neural operator an ideal choice for aggregating/message-passing local information? These seem like good baselines and are currently missing.\n\n- DeepONet gives approximation in a fixed, finite-dimensional, linear subspace of $\\mathcal{V}$, which I think have trouble dealing with discontinuous output functions generally. This is in some sense the motivation for developing nonlinear methods like FNO, NOMAD etc., So this is something that would also be good to compare the proposed algorithm against. \n\n- The design principle of PoU-MoE also seems very tied to the underlying geometry of the problem, which also seems to suggest it might be hard to train if the training data contains problems with very different geometries. This kind of sensitivity should be tested, since it doesn't seem to exist in vanilla DeepONet. \n\n- I think the sentiment that problems that develop shock waves (i.e., sharp spatial gradient) are hard to capture also depends on one's frame of reference. I would think if one has the knowledge that this is the case, it would be a good idea to e.g., picks a Wavelet basis to work with in the first place, since it admits highly sparse representation, rather than a naive DeepONet trunk.\n\nMinor Comments:\n\n- Eqn (2) and (6), left part of the norm is missing.", "broader_impact_concerns": "No concerns involved.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received detailed comments on both the method and presentation, and the authors have made a major revision to address these concerns. After the rebuttal, the paper received mixed recommendations, but overall, the reviewers were satisfied with the revisions. The main concern was the lack of exploration of certain future directions, such as automatic partitioning. I believe these are valuable suggestions that the authors could consider in future work. However, given that the experiments already demonstrate strong performance and the claims are well supported, I recommend acceptance.

Very minor comments: 

(1) the notation for $G(u,y)$ is not consistent in eq. (1), eq. (2) and many other places.

(2) does eq. (2) hold for any $u$? The norm in eq. (2) is not defined.

(3) period in equation (6) should be a comma.

Given these, the authors may want to proofread the paper before submitting the final version.
Decision: Accept (poster)