HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork
Authors: Jae Yong Lee
SungWoong CHO
Hyung Ju Hwang
Venue: ICLR 2023 poster (2023)
Keywords: Hypernetwork
Operator learning
Deep operator network
DeepONet

Abstract: Fast and accurate predictions for complex physical dynamics are a big challenge across various applications. Real-time prediction on resource-constrained hardware is even more crucial in the real-world problems. The deep operator network (DeepONet) has recently been proposed as a framework for learning nonlinear mappings between function spaces. However, the DeepONet requires many parameters and has a high computational cost when learning operators, particularly those with complex (discontinuous or non-smooth) target functions. In this study, we propose HyperDeepONet, which uses the expressive power of the hypernetwork to enable learning of a complex operator with smaller set of parameters. The DeepONet and its variant models can be thought of as a method of injecting the input function information into the target function. From this perspective, these models can be viewed as a special case of HyperDeepONet. We analyze the complexity of DeepONet and conclude that HyperDeepONet needs relatively lower complexity to obtain the desired accuracy for operator learning. HyperDeepONet was successfully applied to various operator learning problems using low computational resources compared to other benchmarks.

Published as a conference paper at ICLR 2023
HYPERDEEPONEt: LEARNING OPERATOR WITH COM- PLEX TARGET FUNCTION SPACE USING THE LIMITED RESOURCES VIA HYPERNETWORK
Jac Yong Lee"
Sung Woong Cho & Hyung Ju Hwang
Center for Artificial Intelligence and Natural Sciences Korea Institute for Advanced Study
Department of Mathematics
Pohang University of Science and Technology Pohang. 37673, Republic of Korea {swcho hiwwang))posseeeeeonnooe @postech. ac. kr
Scoul, 02455, Republic of Korea [jaeyong] @kias.ii..... re.kr kr
ABSTRACT
Fast and accurate predictions for complex physical dynamics are a significant chal- lenge across various applications. Real-time prediction on resource-constraing hardware is even more crucial in real-world problems. The deep operator network (DeepONet) has recently been proposed a framework for learning nonlinear mappings between function spaces. However, the DeepONet requires many pa- rameters and has high computational cost when learning operators, particularly those with complex (discontinuous or non-smooth) target functions. This study proposes HyperDeepONett, which uses the expressive power of the hypernetwork to enable the learning of a complex operator with smaller set of parameters. The DeepONet and its variant models can be thought of as method of inject- ing the input function information into the target function. From this perspec- tive, these models can be viewed as a particular case Nepertee We analyze the complexity of DeepONet and conclude that HyperDeeePONet needs relatively lower complexity to obtain the desired accuracy for operator learning. NyperDeepONet successfully learned various operators with fewer computational resources compared to other benchmarks.
INTRODUCTION
Operator learning for mapping between infinite-dimensionall function spaces is a challenging prob- lem. It has been used in many applications, such as climate prediction (Kurth et al., 2022) and fluid dynamics (Guo et al. 2016). The computational efficiency of learning the mapping is still impor- tant in real-worid problems. The target function of the operator can be discontinuous complicated dynamic systems. In this case, balancing model complexity and cost for computational time is a core problem for the real-time prediction on constre-constrraieed hardware (Choudhary et al. 2020 Murshed et al. (021).
sharp for
Many machine learning methods and deep learning-based architectures have been successfully de- veloped to learn a nonlinear mapping from an dinnnte-ddmenionall Banach space to another. They focus on learning the solution operator of some partial differential equations (PDEs), e.g., the initial or boundary condition of PDE to the corresponding solution. AAnandkumar et al. (2019) proposed an iterative neural operator scheme to learn the solution operator of PDEs.
Simultaneously, Lu et al. 2019 2021) proposed a deep operator network (DeepONet) architecture based on the universal operator approximation theorem of Chen & Chen (1995). The DeepONet consists of two networks: branch net taking an input function fixed finite locations, and trunk net taking a query location of the output function domain. Each network provides the p outputs. The two p-outputs are combined as linear combination (inner- product)) approximate the underlying operator, where the branch net produces the coefficients (p-coefficients) and the trunk net produces the basis functions (p-basis) of the target function.
"These authors contributed equally. corresponding author

---

Published as a conference paper at ICLR 2023
While variant models of DeepONet have been developed to improve the vanilla DeepONet, they still have difficulty approximating the operator for a complicated target function with limited computa- tional resources, Lanthaler et al. 20222) and and Kovachki et al. (2021b) pointed out the limitation of lin- ear approximation in DeepONet. Some operators have a slow spectral decay rate of the Kolmogorov n-width, which defines the error of the best possible linear approximation using an n o--imensiinal space. A large n is required to learn such operators accurately, which implies that the DeepONet requires large number of basis p and network parameters for them.
Hadorn 2022)
investigated the behavior of DeepONet, to find what makes it challenging to detect the sharp features in the target function when the number of basis p is small. They proposed Shift- Neett-DeepONet by adding two neural networks to shift and scale the input function. |Venturi & Casey (2023) also analyzed the limitation of DeepONet via singular value decomposition (SVD) and proposed a flexible DeepONet (flexDeepONet), adding pre-net and an additional output in the branch net. Recently, to overcome the limitation of the linear approximation, [Seidman et al. (2022) proposed a nonlinear manifold decoder (NOMAD) framework by using neural network that takes the output the branch net as the input along with the query location. Even though these methods reduce the number of basis functions, the total number of parameters in the model cannot be decreased. The trunk net still requires many parameters to learn the complex operators, especially with the complicated (discontinuous non-smooth) target functions.
In this study, we propose a new architecture, HyperDeepONerr which enables operator learning, and involves complex target function space even with limited resources. The HyperDeepONett uses a hypernetwork, as proposed by Ha et al. (0177). which produces parameters for the target network. Waan et al. (2022) pointed out that the final inner product in DeepONet may be inefficient if the information of the input function fails to propagate through a branch net. The hypernetwork in HyperDeepONet transmits the information of the input function to each target network's 's parameters. Furthermore, the expressivity of the hypernetwork reduces the neural network complexity by sharing the parameters (Galanti & Wolf 2020). Our main contributions are as follows.
We propose a novel HyperDeepONet using a hypernetwork to overcome the limitations of DeepONet and learn the operators with complicated target function space. The DeepONet and its variant models are analyzed primarily in terms of expressing the target function neural network (Figure 4 These models can be simplified versions of our general HyperDeepONet model (Figure
We analyze the complexity of DeepONet (Theorem 2 and prove that the complexity of the HyperDeepONet is lower than that of the DeepONet. We have identified that the Deep- ONet should employ a large number of basis to obtain the desired accuracy, so it requires numerous parameters. For variants of DeepONet combined with nonlinear reconstructors, we also present lower bound for the number of parameters in the target network.
The experiments show that the HyperDeepONet facilitates learning an operator with a small number of parameters in the target network even when the target function space is compli- cated with discontinuity and sharpness, which the DeepONet and its variants suffer from. The HeperDeepONet learns the operator more accurately even when the total number of parameters in the overall model is the same.
2 RELATED WORK
Many machine learning methods and deep learning-baas based architectures have been successfully de- veloped to solve PDEs with several advantages. One research direction is to use the neural network directly to represent the solution of PDE ( & Yu 2018 Sirignano & Spiliopoulos, 2018). The physics-informed neural network (PINN), introduced by Raissi et al. (2019), minimized the residual of PDEs by using automatic differentiation instead of numerical approximations.
There is another approach to solve PDEs, called operator learning. Operator learning aims to learn a nonlinear mapping from an dimensioaa Banach space another. Many studies utilize the convolutional neural network to parameterize the solution operator of PDEs in various applications (Guo et al. 2016;; Bhatnagar et al.
[2019; Khoo et al. 2021 Zhu et al. 2019 Hwang et al. 2021). The
neural operator (Kovachki et al.
2021b i8 proposed to approximate the nonlinear operator inspired
by Green's is function. Li al." (2021) extend the neural operator structure the Fourier Neural
2

---

Published as a conference paper at ICLR 2023
Operator (FNO) to approximate the integral operator effectively using the fast Fourier transform (FFT), Kovachki et al. (021a) proved the universality of FNO and identified the size of the network.
The DeepONet (Lu et al. 2019; 2021) has also been proposed as another framework for operator learning. The DeepONet has sigmificantly been applied to various problems, such as bubble growth dynamics (Lin al. [02T], hypersonic flows (Mao et al., 2021), and fluid flow (Cai et al. 2021
Lanthaler et al.J (2022) provided the universal ap- proximation property of DeepONet. [Wang et al. (02T)) proposed physics-informed DeepONet by adding residual of PDE as loss function, and Ryck & Mishra (2022) demonstrated the generic bounds on the approximation error for Prasthofer al. (2022) considered the case where the discretization grid of the input function in DeepONet changes by employing the coordinate encoder. Lu et al. (2022) compared the FNO with DeepONet in differ- ent benchmarks to demonstrate the relative perfor- mance. FNO can only infer the output function of an operator the input function in the same grid needs to discretize the output function to use Fast Fourier Transform(FFT). In contrast, the DeepONet can predict from any location.
Output n(t= X1)
Input NNt kg)
Mt= Xx,
Figure I: Example of operator learning: the input function and the output function for the solution operator of shallow water equation
Ha al.) (2017) first proposed hypernetwork, a network that creates a weight of the primary net- work. Because the hypemetwork can achieve weight sharing and model compression, it requires a relatively small number of parameters even as the dataset grows. Galanti Wolf (2020) proved that a hypernetwork provides higher expressivity with low-complexity target networks, [Sitzmann et al. (2220) and Klocek et al. (2019) employed this approach to restoring images with insufficient pixel observations or resolutions. de Avila Belbute- Peres et al. (2021) investigated the relationship be- tween the coefficients of PDEs and the corresponding solutions. They combined the hypernetwork with the PINN's s residual loss. For time-dependent PDE, Pan et al. (2022) designated the time ć as the input of the hypernetwork so that the target network indicates the solution at t. yon Oswald et al.) (2200) devised chunk embedding method that partitions the parameters of the target network; this is because the output dimension of the hypernetwork can be large.
3 OPERATOR LEARNING
3.1 PROBLEM SETTING
The goal of operator learning is to learn mapping from others using a finite pair of functions. Let G 14 + S be a nonlinear operator. where and S are compact subsets of infiitt-dimennsional function spaces U C C(X;; Rd))) and S C(); Rd.) with compact domains X' C Rdd and y C Rdy For simplicity, we focus on the case dy = d, = 1, and all the results could be extended a more general case for arbitrary d, and dz. Suppose we have obser- vations {u,, (u:)) where E U and (w4) E S. We aim to find an approximation Go 2 + S with parameter using the N observations is an important to predict the fluid flow over time according to random initial height of the fluid. To this end, we want find an operator Go. which takes an initial fluid height as an input function and produces the fluid height over time at any location as the output function (Figure T)
function space the
that Go R3 G. For example, in dam break scenario.
u
S
As explained in Lanthaler et al. (022)) the approximator Go can be decomposed into the three components (Figure as
E
R
R
RP
Ge = R o AoE.
(1)
A
First, the encoder E takes an input function u from U to generate the finite-dimensional encoded data in R. Then, the approximator .A is an operator approximator from the encoded data in finite dimension space RT to the other finite- dimensionn space I.. Finally, the reconstructor 'R. reconstructs the output function s(g) (y)) with y y' using the approximated data in IR..
Figure 2: Diagram for the three components for op- erator learning.
3

---

Published as a conference paper at ICLR 2023
3.2 DEEPONET AND ITS LIMITATION
DeepONet can be analyzed using the above three decompositions. Assume that all the input func- tions evaluated at fixed locations 1 X: they called "sensor points." " DeepONet uses an encoder as the pointwise projection {(u) (u(z1), u(z2)..... v(xm)) of the continuous function u, the so-called "sensor values" of the input function u. An intuitive idea is to employ neural network that simply concatenates these m sensor values and query point y as an input to approximate the target function (v)) DeepONet, in contrast, handles m sensor values and query point y separately into two subnetworks based on the universal approximation theorem for the operator (Lu et al.) 2021). See Appendix B for more details. Lu et al. (202T) use the fully connected neural network for the approximator mm - RR.. They referred to the composition of these two maps branch net
B: R",
R. B()) := Ao E(u)
(2)
for any E U. The role of branch net can be interpreted as learning the coefficient of the target function (v)) They use one additional neural network, called trunk net T as shown below.
y R (v) == {Tx(t))E=
for any y/ y. The role of trunk net can be interpreted as learning an affine space that can efficiently approximate output function space C(); ". The functions Ti (y)....TTll become the
p-basis of vector space associated with V and To(y) becomes a point of V. By using the trunk net the r-induced reconstructor R. is defined as
R, R +
R, = Tu((v)
(ATA(V)
(4)
where 3 = (81, 3p) RP. In DeepONet, (v) is restricted to a constant To E that contained in a reconsiructor R. The architecture of DeepONet is described in Figure (b).
Here, the r-induced reconstructor R, is the linear approximation of the output function space. Be- cause the linear approximation R cannot consider the elements in its orthogonal complement, priori limitation on the best error of DeepONet is explained in Lanthaler et al. (2022) as
Un ((w)()) R, Ao E())()) (ddd(uu)
Ak. k>P
where A A2 2 ... are the eigenvalues of the covariance operator I' of the forward measure G Several studies point out that the slow decay rate of the lower bound leads to inaccurate approximation operator learning using DeepONet (Kovachki et al. 202Th] Hadorn] 2022 Lanthaler et al., (022)) For example, the solution operator of the advection PDEs (Sei-) dman et al., 2022, Venturi & Casey) 2023) and of the Burgers' equation (Hadorn) 2022) are difficult to approximate when
using the DeepONet with the small number of basis p.
3.3 VARIANT MODELS OF DEEPONET
How to put information of u E U
Several variants of DeepONet have been developed overcome its limitation. All these models can be viewed from the perspective of parametrizing the target function as a neural network. When we think of the target net- work that receives y as an input and generates an output (y)) the DeepONet and its variant model can be dis- tinguished by how information from the input function injected into this target network Go(u), described in Figure 3 From this perspective, the trunk net in the
Go (4)
Target network
Figure 3: The perspective target network parametrization for operator learning.
the target network cept for the final output, as shown in Figure 4 (a). The output of the branch net gives the weight between the last hidden layer and the final output.
DeepONet be considered
Hadorn 2022 proposed Shift-DeepONet. The main idea that a scale net and a shift net are used to shift and scale the input query position y. Therefore, it can be considered that the information of
4

---

Published as a conference paper at ICLR 2023
Branch
Branch
Sools rart
Stef
bris
5e00
Se00
Trascnet
Teant
Thrget network
Thrget network (c) FlexDeepONet
network
(a) DeepONet
(b) Chift-DeepONet
(d) NOMAD
Figure 4: DeepONet and its variant models for operator learning.
input function generates the weights and bias between the input layer and the first hidden layer, as explained in Figure (b).
Venturi & Casey (2023) proposed FeexDeepONet, explained in Figure (c). They used the addi- tional network, pre-net, to give the bias between the input layer and the first hidden layer. Addition- ally, the output of the branch net also admits the additional output To provide more information on input function u at the last inner product layer.
NOMAD recently developed by Seidman et al. (2022) overcome the limitation of DeepONet. They devise nonlinear output manifold using neural network that takes the output of branch net {5.)%-1 and the query location y. As explained in Figure (d), the target network receives infor- mation about the function u as an additional input, similar to other conventional neural embedding methods (Park et al. 2019 Chen & Zhang 2019 Mescheder et al. 2019).
These methods provide information on the input function only part of the target network. It is a natural idea to use hypernetwork to share the information of input function u to all parameters of the target network. We propose a general model HyperDeepONet (Figure which contains the vanilla DeepONet, FlexDeepONet, and Shift-DeepONel, as a special case of the HyperDeepONet.
PROPOSED MODEL: HYPERDBEPONET
4.1 ARCUITECTURE OF HYPERDEEPONET
The HyperDeepONNet structure is described in Figure The encoder E and the approximator AA are used, simi- lar to the vanilla DeepONet. The proposed structure re- places the branch net with the hypernetwork. The hyper- network generate all parameters of the target network. More precisely, we define the hypernetwork h as
u ((xg)
Hyper-net
u(x,,)
W
W
Ge (r)
ha for any u U. Then, h(u) θ E Rp is network parameter the target network, which is used in recon- structor for the HyperDeepONet. We define the recon- structor R as
RD, (w) = A Eu)
(6)
Target network
Figure 5: The proposed HyperDeep-
R: RP
RR), R(O)()) NN(;; Q) (7)
ONet structure
where θ = b) E R2, and NN denotes the target network. Two fully connected neural networks are employed for the hypernetwork and target network.
Therefore, the main idea is to use the hypernetoork, which takes an input function and produces the weights the target network. It can be thought of as weight generator for the target network. The hypernetwork determines the all parameters of the target network containing the weights be- tween the final hidden layer and the output layer. implies that the structure of eeperDeepONet contains the entire structure of DeepONet. As shown in Figure 4 (b) and (c), Shift-DeepONe and FlexDeepONet can also be viewed as special cases of the HyperDeepONet, where the output of the hypernetwork determines the weights or biases of some layers of the target network. The outputs of
5

---

Published as a conference paper at ICLR 2023
the hypernetwork determine the biases for the first hidden layer in the target network for NOMAD in Figure (d).
4.2 COMPARISON ON COMPLEXITY OF DEEPONET AND HYPERDEEPONET
In this section, would like to clarify the complexity of the DeepONet required for the approxi- mation A and reconstruction R based on the theory in Galanti & Wolf (2020). Furthermore, we will show that the HyperDeepOONet entails relatively lower complexity than the DeepONet using the results on the upper bound for the complexity of hypernetwork (Galanti & Wolf 2020).
4.2.1 NOTATIONS AND DEFINITIONS
Suppose that the pointwise projection values (sensor values) of the input function u is given as E (u) = (u(x1): u(z2),..., u (((m)). For simplicity, we consider the case y [-1,1] and E(u) E
|- 1, 1mm For the composition R A RT + IR), we focus on approximating the mapping
O: IR ay - IR, which is defined follows:
((E()), == (R ((E())))v), for E [-1, E()) E [-1,1]]".
(8)
The supremum norm || is defined as max [M(y)]]- Now, we introduce the Sobolev space Wra, which a subset of C* ([-[11,
1, 1]"; R). For r,n E E,
Waa = h 1-111 1 R
|||l] = 11 +
Σ 1S
1 s $1)
where D denotes the partial derivative of h with respect to multi-index k E (Nu (0))%, We assume that the mapping O lies in the Sobolev space Wr,
For the nonlinear activation a, the class of neural network represents the fully connected neural network with depth l and corresponding width (h1 n. h22 11++1), where W E ERh, x Rh4s and b; E 1g/ denote the weights and bias of the i-th layer respectively.
F= (f [f:[-1,1]" + [ [W, bl) W o(wwki....oW'
oow' o y + b-) + b- +0
Some activation functions facilitate an approximation for the Sobolev space and curtail the com- plexity. We will refer to these functions universal activation functions. The formal definition can be found below, where the distance between the class of neural network F and the Sobolev space Wr,, is defined by dr,a) := supyew... fnl [Jʃ //l Most well-known activa- tion functions universal activations that infinitely differentiable and non-polynomial in any interval (Mhaskar) 1996). Furthermore, Hanin Sellke (2017) state that the ReLU activation is also universal.
Definition 1. (Galanti & Wolf 2020) (Universal activation). The activation function a is called universal if there exists class of neural network F with activation function such that the number of parameters of F is O(e -n/r) with d(F;Wr,a) S eor all r, E N.
We now introduce the theorem, which offers guideline the neural network architecture for operator learning. It suggests that if the entire architecture can be replaced with a fully connected neural network, large complexity should be required for approximating the target function. It also verifies that the lower bound for a universal activation function is a sharp bound on the number of parameters. First, we give an assumption to obtain the theorem.
Assumption 1. Suppose that F and Wr represent the class of neural network and the target func- tion space approximate, respectively. Let F' be a neural network class representing a structure in which one neuron is added rather than F. Then, the followings holds for all sb Wrn not contained in F.
inf /l - 1x inf IIʃ FEF
For r 0. Galanti & Wolf (2020) remark that the assumption is valid for 2-layered neural networks with respect to the L' norm when an activation function a is either a hyperbolic tangent or sigmoid function. With Assumption I. the following theorem holds, which is a fundamental approach to identifying the complexity of DeepONet and its variants. Note that real-valued function g e L L'R)) is called bounded variation if its total variation suP Su (()) (x))dr is finite.

---

Published as a conference paper at ICLR 2023
Theorem 1. (Galanti & Wolj 2020 Suppose that F is class of neural networks with piecewise C (R) activation function a R + R of which derivative o' is bounded variation. If any non- constant & W does not belong to F, then d(F; W w) S implies the number of parameters in F should be S2(==n/r)
4.2.2 LOWER BOUND FOR THE COMPLEXITY OF THE DEEPONET
Now, we provide the minimum number of parameters in DeepONet. The following theorem presents a criterion on the DeepONet's complexity to get the desired error. It states that the number of required parameters increases when the target functions are irregular, corresponding to small r. FDeepONer [5, T) denotes the class of function in DeepONet, induced by the class of branch net B and the class of trunk net T.
Theorem 2. (Complexity of DeepONet) Let R - R be a universal activation function in C" (R) such that and a' bounded. Suppose that the class of branch net B has bounded Sobolev norm (i.e., || S 11, v/ E 13). Suppose any non-constant v) E We, does not belong to any class of neural network. In that case, the number of parameters in the class of trunk net T is Me5-dy/ -y//r) when d( (nDDoooone (B, T); Wr,d, +m) S.
The core of this proof showing that the inner product between the branch net and the trunk net could be replaced with a neural network that has a low complexity (Lemma Therefore, the entire structure of DeepONet could be replaced with a neural network that receives (E(u), y] Rdy+m as input. It gives the lower bound for the number of parameters in DeepONet based on Theorem The proof can be found in Appendix C.I
The analogous results holds for variant models of DeepONet. Models such Shift-DeepONet and flexDeepONet could achieve the desired accuracy with a small number basis. Still, there was a trade-off in which the first hidden layer of the target network required numerous units. There was no restriction on the dimension of the last hidden layer in the target network for NOMAD, which uses fully nonlinear reconstruction. However, the first hidden layer of the target network also had to be wide enough, increasing the number of parameters. Details can be found in Appendix C.2
For the proposed HyperDeepONet, the sensor values E(u) determine the weight and bias of all other layers as well as the weight of the last layer of the target network. Due to the nonlinear activation functions between linear matrix multiplication, it is difficult to replace HyperDee with single neural network that receives [E(u), 3] E R. as input. Galanti & Wolf (2020) state that there exists a hypemetwork structure (HyperDecpONet) such that the number of parameters in the target network is O(e-d//r). It implies that the HyperDeepONet reduces the complexity compared to all the variants of DeepONet.
EXPERIMENTS
In this section, we verify the effectiveness of the proposed model HyperDeepONet to learn the operators with a complicated target function space. To be more specific, we focus on operator leamiing problems in which the space of output function space is complicated. Each input function u, generates multiple triplet data points (u,, y. (v)) for different values of y. Except for the shallow water problem, which uses 100 training function pairs and 20 test pairs, use 1,000 training input-output function pairs and 200 test pairs for all experiments.
For the toy example, we first consider the identity operator u4 + u4. The Chebyshev polyno- mial is used as the input (=output) for the identity operator problem. The Chebyshev polynomials of the first kind Ti of degree 20 can be written as u, E {E (x))c E [-1/4, 1/4]] with random d sampling c from uniform distribution U|-1/4, 1 1/4). The differentiation operator G u; + de u; considered for the second problem. Previous works handled the anti-derivative operator, which makes the output function smoother by averaging (Lu et al. 2019 2022). Here, choose the dif- ferentiation operator instead of the anti-derivative operator to focus on operator learning when the operator's output function space is complicated. We first sample the output function ((u) from the above Chebyshev polynomial of degree 20. The input function is generated using the numerical method that integrates the output function.
7

---

Published as a conference paper at ICLR 2023
Model DeepONet
Shift
Flex
NOMAD
Hypertours)
Identity
0.578++0.003 0.559+00.001
0.777=0.018 0.624#0.015
0.678=0.062
0.578=000020
0.036=0.005
Diffcrentiation
0.562=0.016
0.558++0.003
0.127=00.033
Table 1: The mean relative L test emror with standard deviation for the identity operator and the differentiation operator. The DeepONet, its variants, and the HyperDeepONet use the target network dy-20-20- 20-10-10-1 with tanh activation function. Five training trials performed independently.
imput utx) and output doul
[B) Prodiction using DespOnet S variants
[c] Prediction using Hyper HypprDet
Beppe hireton
GeeoOnet
Shit SeepChet
05 120 H03
RONAD
M0 353
120
Figure 6: One test data example of differentiation operator problem.
Finally, the solution operators of PDEs are considered. We deal with two problems with the complex target function in previous works (Lu et al., 2022 Hadorn 2022). The solution operator of the advection equation considered mapping from the rectangle shape initial input function to the solution w(t, z) at f = 0.5, i.e., G : w(0, x) + x).. We also consider the solution operator of Burgers' equation which maps the random initial condition to the solution w(t, x) at 1. i.e., G w(0, .) + c0(1,20 .²). The solution of the Burgers' equation has a discontinuity in a short time, although the initial input function is smooth. For a challenging benchmark, we consider the solution operator of the shallow water equation that aims to predict the fluid height l(t,, 21, x2) from the initial condition (000, x1, x2). i.e., G : h(O, æ1, x2) + f((t, I1, x22 (Figure IT. In this case, the input of the target network is three dimension (t., 21, 22). which makes the solution operator complex. Detail explanation is provided in Appendix E
Expressivity of target network. We have compared the expressivity of the small target network using different models. We focus on the identity and differentiation operators in this experiment. All models employ the small target network 20--20-20-10-1 with the hyperbolic tangent activation function. The branch net and the additional networks (scale net, shift net, pre-net, and hypernetwork) also use the same network size as the target network for all five models.
Table T shows that the DeepONet and its variant models have high errors in learning complex oper- ators when the small target network is used. In contrast, the HyperDeepONet has lower errors than the other models. This is consistent with the theorem in the previous section that HyperDeepONet can achieve improved approximations than the DeepONet when the complexity of the target net- work is the same. Figure [ shows prediction on the differentiation operator, which has highly complex target function. The same trends are observed when the activation function or the number of sensor points changes (Table5) and the number of layers in the branch net and the hypernetwork vary (Figure II].
Same number of learnable parameters. The previous experiments compare the models using the same target network structure. In this section, the comparison between the DeepONet and the HyperDeepONet is considered when using the same number of learnable parameters. We focus the solution operators of the PDEs.
Branch net (Hypereework)
Targer
N Param
Kel emnoe
DeepONet
150
d, 25--6-666666-666-
274K
Advection
Hyper(ouns)
7w-70-00-70-70-20
4,-33-33-33-33-
268K
0.00-88000000
c. <Hyyperours)
1ww.12--18-20 128. 128- 128. 1024
d, 4,-256.2222 256-266666222 256-2266--2 256.1
208K
0.0043-44.004
Deep
0o-128-12N-128-
d,-124-128--
0.1391
Burgers
Hyper(ouns)
mm-66-66-66-66-66--6
4,-20-20020-20011
114K
0.0196:10,0011
e- Hyper(ouns)
1--66-66-6-66666--11
d,-128-128- 128--128-128--- 1-8--128-11111 128-1
115K
0.0066:00.000
Shallow
Dee
Hyper(ouns)
m-30-30-300
m--10-1100 100-1000
4,-100-100-- 100-
4,,30.30. 30-30-1
107K
101K
0.0148
0.0042
0.0002
Shallow
DeepONet
mm-20-36-10
2, 20-20:20 10:1
6.5
0.0091
2 0.0066
w/ small
Hyperiours)
-10-10-10--0
d1-10-10-10-1
5.7K
0.0209
0.0013
Table 2: The mean relative 2 test eπror with standard deviation for solution operator learning problems, No and #Param denote the number of parameters in the target network and the number of learnable parameters, respectively. Five training trials are performed independently.
8

---

Published as a conference paper at ICLR 2023
Input output
Prediction after 10000
Prediction after 40000
output
Target function
0.2
HyperDeepDNee
(4) Input uS0 output Slol(n)
151 Prediction after 5000 eocchs
151 Prediction after 50000 opochs
35
Cutput
-35
-10
Figure 7: One test data example of prediction on the advection equation (First row) and Burgers' equation (Second row) using the DeepONet and the HyperDeepONet.
For the three solution operator learning problems, eu et al.] (202)) and [Seidman et al. (2022) for DeepONet. First, we use the smaller target network with the larger hypemneworr for the HyperDeepONet
the same hyperparameters proposed in
compare the DeepONet. Note that the vanilla DeepONet is used without the output normalization or the boundary condition enforcing techniques explained in Lu al. (2022) to focus the primary limitation of the DeepONet. More Details are in Appendix E Table shows that the HyperDeepONet achieves a similar or better performance than the DeepONet when the two models use the same number of learnable parameters. The HeperNeepONet has slightly higher error for advection equation problem, but this error is close to perfect operator prediction. It shows that the complexity of target network and the number of learnable parameters can be reduced to obtain the desired accuracy using the HyperDeepONet. The fourth row of Table 2 shows that HepeoNee much more effective than DeepONet in approximating the solution operator of the shallow water equation when the number of parameters is limited. Figure 7 and Figure [2 show that the HyperDeepONet learns the complex target functions in fewer epochs for the desired accuracy than the DeepONet although the HyperDeepONet requires more time to train for one epoch (Table 8)
Scalability. When the size of the target network for the HyperDeepONett is large, the output of the hypernetwork would be high-dimensional (Ha et al., 2017 Pawlowski et al., 2017) so that its complexity increases. In this case, the chunked HyperDeepONNe with a trade-off between accuracy and memory based on the chunk embedding method developed by vvon Oswald al. (2020). It generates the subset of target network parameters multiple times iteratively reusing the smaller chunked hypernetwork The c-HyperDeepONet shows a better accu- racy than the DeepONet and the HyperDeeeONeet using an almost similar number of parameters, as shown in Table 2 However, it takes almost 2x training time and 2~30xx memory usage than the HyperDeepPneet More details on the chunked hypernetwork are Appendix [
(e-HyperDeepONNe)
be used
CONCLUSION AND DISCUSSION
In this work, the HyperDeepONet is developed overcome the expressivity limitation of Deep- ONet. The method of incorporating an additional network and nonlinear reconstructor could not thoroughly solve this limitation. The hypernetwork, which involves multiple weights simultane- ously, had a desired complexity-reducing structure based on theory and experiments.
We only focused on when the hypernetwork and the target network is fully connected neural net- works. In the future, the structure of the two networks can be replaced with a CNN or ResNet, as the structure of the branch net and trunk net of DeepONet can be changed to another network (Lu al.) 2022). Additionally, it seems interesting research a simplified modulation network proposed by Mehta et al. 2022), which still has the same expressivity as HyperDeepONet.
Some techniques from implicit neural representation can improve the expressivity of the target net- work (Sitzmann et al. 2020). Using a sine function as an activation function with preprocessing will promote the expressivity of the target network. We also leave the research on the class of activation functions satisfying the assumption except for hyperbolic tangent or sigmoid functions as a future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
J. Y. Lee was supported by KIAS Individual Grant (AP086901) via the Center for Al and Natural Sciences at Korea Institute for Advanced Study and by the Center for Advanced Computation at Korea Institute for Advanced Study. H. Hwang and S. W. Cho were supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
00165268) and by Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea (overnment(MSSP) (N0.2019---/1906, Artificial Intelligence Graduate School Program (POSTECH))
REFERENCES
Anima Anandkumar, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differ- ential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019. URL /tctpa://epenreviiii
net// Eorum? 1d=fg22FmXFO3
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Pre- diction of aerodynamic flow fields using convolutional neural networks. Comput. Mech., 64
(2):525-545, 2019. ISSN 0178-7675.
doi: 10.0007/50466--099-1774444 URL https:
//doi. 1000://10.0000/00666
0029-01440-
Shengze Cai, Zhicheng Wang, Lu Lu, Tamer A Zaki, and George Em Karniadakis. Deepm&mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:1 10296, 2021.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5939-5948, 2019.
Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A compre- hensive survey on model compression and acceleration. Artificial Intelligence Review, 53(7): 5113-5155, 2020.
Filipe de Avila Belbute- Peres, Yi-fan Chen, and Fei Sha. Hyperpinn: Learning parameterized differ- ential equations with physics-informed hypernetworks. In The Symbiosis of Deep Learning and Differential Equations, 2021.
Weinan E and Bing Yu. The deep Ritz method: deep learning numerical algorithm for solving variational problems. Commun. Math. Stat. 6(1): 2018. ISSN 2194-6701. doi: 10. 0007/540304-018-012
018 0120-117711
Tomer Galanti and Lior Wolf. On the modularity of hy Advances in Neural Information Processing Systems, 33: 14409-1011449 2020.
Xiaoxiao Guo, Wei Li, and Francesco lorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 481 1911-490,22 2016.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetorrss. In International Conference on Learn- ing Representations, 2017. URL htttts::/openreviiiii www..neeoooommmmdro.......
Patrik Simon Hadorn. Shift-deeponet Extending deep operator networks for discontinuous output functions. ETH Zurich, Seminar for Applied Mathematics, 2022.
Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv: 1710.1710.1110 11278, 2017.
Rakhoon Hwang, Jac Yong Lee, Jin Young Shin, and Hyung Ju Hwang. Solving pde-constrained control problems using operator learning. arXiv preprint urXiv:2111.04941 2021.
10

---

Published as a conference paper at ICLR 2023
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric PDE problems with artificial neural networks. European J. Appl. Math., 32(3):221 435, 2021 ISSN 7925, doi: 10.1017/ 50956792522000182. UtLhttps::/
101
Diederik P.
Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. htth: 7/ //axxiv.org/aaaaavv
6980
Sylwester Klocek, łukasz Maziarka, Maciej Wolczyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. Hypernetwork functional image representation. In International Conference on Arti- ficial Neural Networks, pp. 196-510. Springer, 2019.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 202 la
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108..888 2021b.
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators. arXiv preprint MrXiv:2208.054199 2022.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applica- tions, 6(1 6(1):tnac001, 2022
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadeneshelii Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL
https::/<pperewwe
net http:r/rreeeeetttfooooolllgeggg
Chensen Lin, Zhen Li, Lu Lu, Shengze Cai, Martin Maxey, and George Em Karniadakis. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10):104118 2021.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: 19t0.:9900003 109,, 2019.
Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 213)22182229 2021.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineer- ing, 393: 14778, 2022.
Zhiping Mao, Lu Lu, Olaf Marxen, Tamer A. Zaki, and George Em Karniadakis. & for hypersonies: predicting the coupled flow and finite-rate chemistry behind a normal shock using approximation of operators. J. Comput. Phys., 447:Paper No. 1 106988, 24, 2021. doi: 10.1016/j.ec..02220 20221 10698, URL httph::///// //tpi. http:///ooooogg..../ wwt.ooogg............ 1016//..0
ISSN 0021-9991.
jep. 2021. 10698
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEÉ/CVE International Conference on Computer Vision, pp. 14214-14223, 2021.
11

---

Published as a conference paper at ICLR 2023
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460 4470, 2019.
Hrushikesh N Mhaskar. Neural networks for optimal approximation of smooth and analytic func- tions. Neural compulation, 8(1):164-177 1996.
MG Sarwar Murshed, Christopher Murphy, Daqing Hou, Nazar Khan, Ganesh Ananthanarayanan, and Faraz Hussain. Machine learning at the network edge: A survey, ACM Computing Surveys (CSUR), 54(8):1-37, 2021.
Shaowu Pan, Steven Brunton, and J Nathan Kutz. Neural implicit flow: a mesh-agnostic dimen- sionality reduction paradigm of spatio-temporal data. arXiv preprint arX1v:2204003216, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv: 2017.
Michael Prasthofer, Tim De Ryck, and Siddhartha Mishra. Variable-inpur deep operator networks.
arXiv preprint arXiv:2205. 1404, 2022.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. J. Comput. Phys., 378:686-707, 2019. ISSN 0021-9991. doi: 10.1016/j.jcp.2000 201. 10.045. URL
org.i..rgg/aa 1016/ 1tt 2018 10. 045
Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics- informed (and) operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net forumr waraummdd=bbeeeeee 4e wtwp://wwwwweeeettteeeee...
Jacob H Seidman, Georgios Kissas, Paris Perdikaris, and George J Pappas. Nomad: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.03551, 2022.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys., 375:1339 2018. ISSN 0001199911. doi: 10.1016/ 2jcp.2011..88 8.08.029999 08.0229 /RLL:tttss///ooiorrtt 1tt.://..........11.11..111/11.. 10100..111111100
2018 08. 029
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: An extensive benchmark for scientific machine learn- ing. In Thitty-sixth Conference on Neural Information Processing Systems Datasets and Bench- marks Track, 2022.
pttttss://pperreeeee net/
wtt.eeeeweeeeeeeeeeeoooiiiiiaaaa forum?id=ds MkX0OfrK
Simone Venturi and Tiernn Casey. Svd perspectives for augmenting deeponet flexibility and inter- pretability. Computer Methods in Applied Mechanics and Engineering, 403:115718, 2023.
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020. URL
net. Forrrt//oummiim htt.::/eeee..oooooooooooceeeeer SJgwNerKvB
Sifan Wang. Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physics-inie physics-ifformed decponets. Science advances, 7(40)caab86055, 2021.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Improved architectures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2):1-44 2022.
12

---

Published as a conference paper at ICLR 2023
Yinhao Zhu, Nicholas Zabaras, Phacdon- Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for bigh-dimensionnll surrogate modeling and uncertainty quantification without labeled data. J. Comput. Phys., 394:56 81, 2019. ISSN 0021-9991. doi: 10. 2019.05.024. URL http:tttpp...o....o/0000000001// jep 2019 05.024.
13

---

Published as a conference paper at ICLR 2023
A
NOTATIONS
We list the main notations in Table which is not concretely described this paper.
Domain of input function
y
Domain of output function
d,
Dimension 2
dy
Dimension of y
de
Dimension of the codomain of input function
d.
Dimension of the codomain of output function Sensor points
{21.
Fm
m
Number of sensor points Fuclidean space of dimension d Set of functions that has continuous r-th deriva- tive.
Rd
(T(R)
C():
Set of continuous function from y to Rd
1"",-,,1";;
Set of functions from [-1.1]" to R whose r-th partial derivatives continuous.
n = 0(e)
There exists constant C such that n < C
e, Ve 0
((E)
There exists constant C such that > C
e, Ve >
d(C)
n/cc converges to 0 as approaches to 0.
Table Notation
B ORIGINAL STRUCTURE OF DEEPONET
u(xg) uxx2)
b
b2
Branch net
:
ue (x
bp
X
G(u)
t1
Inner
y
Trunk net
t2
Product
:
Figure 8: The structure of DeepONet
For a clear understanding of previous works, we briefly leave a description of DeepONet. In partic- ular, we explain the structure of the unstacked DeepONet in Lu al. (2019) which being widely used in various experiments of the papers. Note that Figure 4 a) represents the corresponding model which called simply DeepONet throughout this paper. The overall architecture of the model is formulated as
R, o ( ... ((r == (B(u(z)), ... (.t......,((.......... ): 0z), ((;;6));
where and 3 are referred to as the trunk net and the branch net respectively. Note that Rr and A, denote the reconstructor and the operator in Section 3.1 For fixed observation points (x1, 2mm) E X the unstacked DeepONet consists of an inner product of branch Net and trunk Net, which are fully connected neural networks. For function the branch net receives pointwise projection values (u(zi),- ... u(xm)) as inputs to detect which function needs to transformed. The trunk net queries location y y of interest where y denotes the domain of output functions.
14

---

Published as a conference paper at ICLR 2023
It was revealed that the stacked DeepONet, the simplified version of the unstacked DeepONet, is a universal approximator in the set of continuous functions. Therefore, the general structure also becomes universal upproximator which enables close approximation by using sufficient number of parameters. Motivated by the property, we focus on how large complexity should be required for DeepONet and its variants achieve the desired error.
C ON COMPLEXITY OF DEEPONET AND ITS VARIANTS
C.I PROOF OF THEOREM NNN DEEPONET COMPLEXITY
The following lemma implies that the class of neural networks is sufficiently efficient to approximate the inner product.
Lemma 1. For the number of basis p E N, consider the inner product function Tp. 1]1, defined by
P xp(a1, ap. b1, bp) = Σ abb = ((a1.'.-,, ,qp), (bi,
,bb)))
For an arbitrary positive I, there exists class of neural network F with universal activation a IR such that the number of parameters of F is O(p +1ptt1/te-1// with inf JEF Il Tplloo S e.
Proof. Suppose that is positive integer and the Sobolev space W2.2 is well defined. First, we would like to approximate the product function [-1, - IR which is defined as
(a, b) = ab.
Note that partial derivatives Dk = for all multi-index k E (0)) such that [k] 2 2. For multi-index k with |k] = Dk contains only term which either or b. In this case, can simply observe that > k|=1 || T I 2- = 2 by the construction of the domain [|1, for πi. And finally,
|| < |lab|l>o I ||a|==||b|l= < 1 1=1,
so that function / should be contained in Wr.2 for any N. In particular, / / lies in W2.2 so that there exists neural network approximation fan in some class neural network F with an universal activation function a such that the number of parameters of F is (((c3p)) -2/20)
O(p -1/),, and
T// - Jnnllo S <//3p,
by Definition I Then the neural network 3f approximates the function गı by an error e/p which can be constructed by adjusting the last weight values directly involved in the output layer of neural network fan-
Finally, construct neural network approximation for the inner product function Ty. Decompose the 2p-22ppmee dip-dnnnsiiona inner product function Tp into p product functions (T,)) which are defined as
Proj, 222r R.
Proj, ap b.. .b,) = mi (ai, by) a,b
for Vi E (1......1 .p). Then each function (T,) could be approximated within an error c/p by neural network NN, which has (tp//( parameters by the above discussion. Finally, by adding the last weight [1, 1,... .1] E RIxp which has input as the outputs of neural networks (NN.) we can construct the neural network approximation NN of T. = Σi= (roje(Tpp) such that the number of parameters is O tppppp///ccll = (1/1)// Class of neural network F. which represents the structure of NN, satisfies the desired property.
Obviously, the statement holds for an arbitrary real which not an integer.
15

---

Published as a conference paper at ICLR 2023
Now assume that O (defined in Eq. (8) lies in the Sobolev space - Then, obtain the following lemma which presents the lower bound on the number basis in DeepONet structure. Note that we apply L. -norm for the outputs of branch net and trunk net which are multi- dimensional vectors.
Lemma 2. Let a R R. be a universal activation function in C" (R) such that o' is bounded variation. Suppose that the class of branch net B has a bounded Sobolev norm (i.e., || S 4, v e B). Assume that supremum norm || lloe the class of trunk net T is bounded by l2 and the number of parameters in T is o(c-o(o (Bd-b/////) If any non-constant dp E W7,dy-m does not belong to any class of neural network, then the number of basis p in T is (((-dv/r) when [FoppOOe (B, T); Wr.dy +m) S
Proof. To prove the above lemma by contradiction, we assume the opposite of the conclusion. Sup- pose that there is no constant C' that satisfies the inequality p > C(e-d./r for Ve > 0. In other words, there exists sequence of DeepONet which has {PaIn= the number of basis with sequence of error {en]m- in IR such that en - and satisfies
Pn I b///r (ie.., Pa = ole,4v/r) with respect to m).
(9)
and
d( (B.. Tn); W Wndy+m) S En.
where B, and Tn denote the corresponding class sequence of branch net and trunk net respec- tively. Then, we can choose the sequence of branch net {3, R" + Rp 100 and trunk net Tn R24 + Rpa 0o n=1 satisfying
[[O(E(u),)) Tp. (E ( I 2c,,, V(E(u), v) 1-1, O Wr.dytm
for the above sequence of DeepONet by the definition of dl (FDeepoNee (B., Ta); Wr.d,tm.
Now, we would like to construct neural network approximations fn for the branch net 8n. By the assumption on the boundedness of the i-th component [[x,], of {3, has a Sobolev norm bounded by /1. In other words, [|B.)]///ll] < 1 and therefore, [3,];/11 / is contained in Wi.m- Since is universal activation function, we can choose neural network approximation [fr]] of [Sn]s such that the number of parameters is ((En/4))=m)) m//) = Oeen """""""", and
[l[f=] [Bn]ill S G./l1
Then, fn = (411m]11-44Alee.. .ll.))p..) becomes neural network approximation of Sn which has Opnn. parameters within an error Gn-
Recall the target function corresponding to m observation Eu)) E []
r by O(E(u), R
which defined Eq. Bb. Then, for (u),, we can observe the following inequalities:
[[O(E(u),)) v) Tr. (J Tn)((u)))))llll (y))) < (I((E(u..)) pT,. (8n (E(u)) Ta(1)) 1l + 1Tp,. (Bn (8 (vx) (w)))l
S 2e, + 6, (v) I (2 12),
by the assumption the boundedness of TT Now would like to consider the sequence of neural network which is an approximation of inner product between pa, dimensional vector in [-1, 1 Note the following inequality
ll
W Ilf.(((w) - (E(W))) + (E())) $ nn (E(n)))
S 6n + 1 < 2h,
with || l2 for large n. It implies that fn(E()))/21 and (x(2//12 lie in [-1, By Lemma there exists a class of neural network H. such that the number of parameters is Opp 141/2dgr -1/2dy") and,
inf || - Tpp 1l10 S 'n heR,
16

---

Published as a conference paper at ICLR 2023
where Tp., is the inner product corresponding Pa. dimensional vector. Choose neural network hn [-1, 1 IR such that I/,, Tp. al I n.n. Then, by the triangular inequality,
|[O(E(u),)) 22 fn (E(u)) Tr. (r)))
Tw)))l) -0ll Tn (1)/
+ 2T (f,(E(()) (2 22,22 = (2 (2+ 44212). (10)
Finally, we compute the number of parameters which is required to implement the function 24, 2412hn (E)) Tx (v)/22). The only part that needs further consideration is scalar multi- plication. Since we need one weight to multiply a constant with one real value, three scalar multi- plications
21, 22212h (n(hn(E(()))2, (il (x((()) (2( and (2) H Tn (1)/222
require 1, Pn. Pn parameters respectively. Combining all the previous results with the size of trunk net, the total number of parameters is obtained in the form of
0(1 + 2p + Pn
1+1/2dg² "- + Pn m//) + o( = o(c,, (((c tm))/r))
since the initial assumption ( the number of basis gives the following inequality.
Pn 6n -1/2dy"
1+1/2dgr
pn m < Pn (pr
-m/r
(p// 'n
1/2d,r -1/2dgr + 6. m,///r)
(E, 1/2d2 + tnn m//) dytr
$
- n
11
1
20, -(dy+m)/ 22
mm n
On the one hand, the sequence of function (J (E(() (1,
is an sequence of
approximation for (EE(u),)) y) within corresponding sequence of error fc,, (2 + le 41112) = Denote the sequence the class of neural networks corresponding to the sequence of the function 12h, (())(((())))))) Tn ( by ) By the assumption, Theorem
1 implies the
number of parameters in {F === is (2((,, (2 +2+ l2 + (1, = 2i(,n (d, +m) /////) There- fore, the initial assumption would result a contradiction so the desired property valid. Li
Note that the assumption the boundedness of the trunk net could be valid if the bounded universal activation function a. Using the above results, we can prove our main theorem, Theorem
2
Proof of Theorem Denote the number of parameters in T by Ny. Suppose that there is no con- stant C satisfies the inequality Ny 2 Cle C'-ddy/r Ve > 0. That is, there exists sequence of DeepONet with the corresponding sequence of trunk net class [Tm and sequence of error { such that + 0, and it satisfies
NT. -e-d//r (i.e.,NTT =(E du/t) with respect to to.).. 11.
Note that the above implies Ny,, = olen (m/+))/") On the one hand, the following inequality holds where B, denotes the corresponding class sequence of branch net.
d(FDeepuNe (B,, Tn); Wr,d,+m) S En-
Since is bounded, T, consists of bounded functions with respect to the supremum norm. There- fore, if we apply the Lemma with respect to n, the number of basis p. should 22c,,//r) Since Pa is also the number of output dimensions for the class of trunk net T,, the number of parameters Ta should be larger than Pa S2 dd/r). This leads to a contradiction.
Finally, we present lower bound the total number of parameters of DeepONet, considering the size of the branch net. Keep in mind that the proof of this theorem can be applied to other variants of DeepONet.
17

---

Published as a conference paper at ICLR 2023
Theorem 3. (Total Complexity of DeepONet) Let a R R be a universal activation function in C" (R) such that a and o' are bounded. Suppose that the class of branch net B has bounded Sobolev norm (i.e., || S h, VB B). any non-constant & Wr.n does not belong any class of neural network, then the number of parameters in DeepONet is (d(e-(dtmm// for any R when d( DeeDDooaaa T); Wr,d++m) e.
Proof. For a positive e < [, suppose that there exists class branch net and trunk net 7 such that di DeepONea [3, T); Wr.dy+m) I By the boundedness of there exists constant /2 which is the upper bound on the supremum norm || bloe of the trunk net class T. Let us denote the number of parameters in DeeeOONet by Nimponse Using the Lemma T replace DeepONet's inner products with neural networks as in the inequality (TO). we can construct a class of neural network F such that the number of parameters F is (
Fireptnaa + +p +1/t 1/t)) and,
d(F;
S (1 + 112)6.
Suppose that Fossosa o(E (4, Then, by Theorem p2t1/le- 1/t should be Q(c-(yy (tm)/rr) Since f can be arbitrary large, the number of basis p should be 22(e ((c-(d, for any R >
L
C.2 COMPLEXITY ANALYSIS VARIANTS OF DEEPONET.
We would like to verify that variant models of DeepONet require numerous units in the first hidden layer of the target network. Now we denote the class of Pre-net Shift-DeepONet and flexDeepONet by P. The class of Shift-DeepONet and flexDeepONet will be written as DehittDerpON (P, B, T) and FRexDespONer (P, B, T) respectively. The structure of Shift-DeepONet can be summarized as follows. Denote the width of the first hidden layer of the target network by
w. We define the pre-net as p [PI, p2] Rnm + Bwx(d,+1 +1) where P Rm + Rwxdy and P2 I R" the branch net as B T - IRP and the trunk net as T R RR. The
Shift- Seftt-DeepONet Dshitt-ecpONaa(p, 3, r) is defined as
Desin- (p. (E 3) == (B(() (2((())))))))))))
where 4 is defined in Eq.
We claim that it does not improve performance for the branch net to additionally output the weights on the first layer of a target network. The following lemma shows that the procedure can be replaced by small neural network structure.
Lemma 3. Consider a function Rdy((r+1) + R which defined below.
Φ ((11," Fdy,
Ld, 1) Bdywr .1. Vdy)
dy Fidd- 1ywtilhi
Fells
(11)
For any arbitrary positive I, there exists class of neural network FF with universal activation
R R such that the number of parameters of F is Owwd)+1// 1/t -1/t) with fe f II &1 S
construct sequence of neural network {i which is an c-approximation of the inner product with O 6 c1/1t parameters. If we combine all of the w
Proof. Using the Lemma
approximations, we get the desired neural network.
Now present the lower bound on the number of parameters for Shift-DeepONet. We derive the following theorem with an additional assumption that the class trunk net is Lipschitz continuous. The function d + IP is called Lipschitz continuous if there exists constant C' such that
(T]) ( l1 < Cl 32ll2.
For the neural network f. the upper bound of the Lipschitz constant for f could be obtained as [ where L. is the Lipschitz constant of a and the norm || ||1 denotes the matrix
18

---

Published as a conference paper at ICLR 2023
norm induced by vector 1-norms. We impose constraints the upper bound of the weights, which consequently enforce affine transformation W to be bounded with respect to the L' norm. Therefore, we guarantee the Lipschitz continuity of the entire neural network in this way.
We would like to remark on the validity of the weight assumptions in the theorem since the bounded assumptions of the weight may be possible reason for increasing the number of parameters. How- ever, the definition of Sobolev space forces all elements to have the supremum norm lo less than 1. It may be somewhat inefficient to insist on large weights for approximating functions with limited range.
Theorem 4. Let a R. + I be a universal activation function in C" (TR) such that and o' are bounded. Suppose that the class of branch net B and pre-net P has a bounded Sobolev norm (i.e., || (, VB E B, and ||ll I 13, Vp E P) and any neural network in the class of trunk net T is Lipschitz continuous with constant 12. If any non-cone non-constant tp E Wrn does not belong to any class of neural network, then the number of parameters in T is ((c-d//t) when d( shp Deepooel (P,, B, T); Wedd, S.
Proof. Denote the number of parameters in T by Ny. Suppose that there exists sequence of pre- net [pn)== branch net {,,n J== and trunk net [Tnn== with the corresponding sequence of error {en) 100 n=1 such that 6n. + 0 and.
o
and
sup WeWeddy+m
Depp In Tn) 1011000 S En.
The proof can be divided into three parts. Firstly, we come up with a neural network approximation NNN of Pr. of which size is O(w En approximation of & using the Lemma Finally, the inner product Tp., (8n, Tr) is replaced with a neural network as in 10 of Lemma
m///) within an error en. Next, construct a neural network
Since all techniques such as triangular inequality are consistent with the previous discussion, we will briefly explain why additional Lipschitz continuity required for the trunk network, and omit the details. Approximating the Pre-Net of Shift DeepOnet, which is not in DeepOnet, inevitably results in an error in the input of the trunk net. We are reluctant to allow this error change the output of the trunk net significantly. In this situation, the Lipschitz continuity provides the desired result. O
For d, = 1, the additional rotation is only multiplying by 1 or -1 Since the weight and bias of the first layer alone can cover the scalar multiplication, flexDeepONet has the same properties as Deet--eeepOet in the above theorem.
Theorem 5. Consider the case dy Let R - ER be a universal activation function in (TR such that a and o' are bounded. Suppose that the class of branch net and pre-net P has a bounded vB E B, and ||ell; 1,, vp P), and any neural network in the class of trunk net T is Lipschitz continuous with constant l2. If any non-constant vi E Wr. does not belong to any class of neural network, then the number of parameters in T is Di (6(--d//) when d(F (B. T); Wr.d,+m) S..
Sobolev norm(i.e., ||]|
Proof. The main difference between flexDeepONet and Shift-DeepONet, which is not mentioned earlier, is that the branch net affects the bias of the output layer. However, adding the values of the two neurons can be implemented in neural network by adding only one weight of value 1 for each neuron, so all of the previous discussion are valid.
In fact, NOMAD be substituted with the embedding method handled by Galanti & Wolf (200 Suppose that the branch net of NOMAD is continuously differentiable. Let's also assume that the Lipschitz constant of branch net and trunk net is bounded. We would like to briefly cite the relevant theorem here.
Theorem 6. (Galanti & Woll (020h) Suppose that is a universal activation in Ca C²(R) such that d' is a bounded vartation on R. Additionally, suppose that there is no class of neural network that can represent any function in Wi other than a constant function. If the weight on the first layer of target network NOMAD is bounded with respect to L--norm, then d(N; Wi. dm)) S implies the number of parameters in N is Q(E -mnnld m where N denotes the class of function contained target network of NOMAD.
19

---

Published as a conference paper at ICLR 2023
D CHUNKED EMBEDDING METHOD
The HyperDeepONer may suffer from the large complexity of the hypernetwork when the size of the target network increases. Although even small target network can learn various operators with proper performance, a larger target network will be required for more accurate training. To take into account this case, we employ a chunk embedding method which is developed by von Oswald et al. (2020). The original hypernetwork that the complexity of hypernetwork could be larger than the complexity of the target network. Such problem can be overcome by using hypernetwork with smaller outputs.
designed generate all of the target network's 's weights so
y
Target
Network
G(u)
Parameter Group
Parameter
Parumeter
Parameter Group N:
Group 2
Group
fn
fn
fn
fn
Λ
(u(x)) x
(u({)) 22
[u(x:)] Xs
[u(r;)) Rye
Figure 9: Chunk Embedding Method
More specifically, Figure 9 describes how the chunk embedding method reduces the number of learnalle parameters. First, they partition the weights and biases of the target network. The hyper- network then creates the weights of each parameter group by using the sensor values {u(x;)) ) with a latent vector Zj. All groups share the hypernetwork factor of the number of groups. Since the latent vectors (N, learn the characteristics of each group during the training period, the chunked embedding method preserves the expressivity of the hypernetwork. The chunked architecture a universal approximator for the set continuous func- tions with the existence of proper partitions (Proposition in yon Oswald et al. (2020)). We remark that the method can also generate the additional weights and discard the unnecessary ones when the number of the target network's parameters is not multiple of No. which is the number of group.
that the complexity decreases by
E EXPERIMENTAL DETAILS
DeepONet
NOMAD
Hyper(ours)
Identity
0.0005
0.0002
0.0005
0.0001
0.0001
Differentiation
0.0005
0.0002
0.0005
0.0001
0.0001
Advection
0.0005
0.0001
0.0001
0.0002
0.0005
Burgers
0.0001
0.0001
Shallow
0.0001
0.0005
0.0001
0.0005
Table 4: Setting of the decay rate for each operator problem
In most experiments, we follow the hyperparameter setting Lu et al. (019 2021 2022). We use ADAM in Kingma & Ba (2015) In all experiments, an InverseTimeDecaa Tema scheduler was used, and the step size was fixed to 1. In the experiments of identity and differentiation operators, grid search was performed using the sets 0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005 for decay rates. The selected values of the decay rate for each model can be found in Table
optimizer with learning rate of le 3 and zero weight decay.
20

---

Published as a conference paper at ICLR 2023
(a) Input uDx) and outpuc (tu)o)
(b) Prediction using DeepOnet variants
(c) Prediction using HyperDeepONet
3.0
20
20
0104
03-
lerget function
ROM
4-
Shift DeesONet
Deepore:
RexDeeepltet
0.d 825
6.75
-075 -053
825
Hype DeperDeepONet
Figure 10: One test data example of differentiation operator problem.
E IDENTITY
As in the text, we developed an experiment to learn the identity operator for the 20th-order Cheby- shev polynomials (Figure TO). Note that the absolute value of the coefficients of all orders less than or equal to 144. We discretize the domain [-1, 1] with a spatial resolution of 50. For the experi- ments described in the text, we construct all of the neural networks with tanh. We use 1,000 training pairs and 200 pairs to validate our experiments. The batch size during the training is determined to be 5,000, which is one-tenth the size of the entire dataset.
E.2 DIFFERENTIATION
In this experiment, set functions whose derivatives 20th-order Chebyshev polynomials as input functions. As mentioned above, all coefficients of the Chebyshev polynomial are between -1/4 and 1/4. We the 100 uniform grid the domain [-1, 1]. The number of training and test samples is 1,000 and 200, respectively. We use batch size 10,000, which is one-tenth the size(100,, 000 100 1, 000) of the entire dataset.
E.3 ADVECTION EQUATION
We consider the linear advection equation on the torus T := IR/Z follows:
Ow at
+
Ow
da = 0.
(12)
w(x 0) wo(), x E T,
where e is a constant which denotes the propagation speed of w. By constructing of the domain as T. we implicitly assume the periodic boundary condition. In this paper. we consider the case when e = 1. Our goal is to learn the operator which maps wo(x) w(0,x)) to w)..,, We use the same data as in Lu al. (2022). We discretize the domain [0. 1] with spatial resolution of 40. The number of training samples and test samples is 1,000 and 200, respectively. We use the full batch for training so that the batch size 40 1, 000 = 40, 000.
E.4 BURGERS' EQUATION
We consider the ID Burgers' equation which describes the movement of the viscous fluid
+ v 2a Du w(I 0) = (x)
dw ăt u. aw Sx
(a, t) E (0, 1) x (0, I E 10,
(13)
where wp is the initial state and v is the viscosity. Our goal is to learn the nonlinear solution operator of the Burgers' equation, which is a mapping from the initial state (n(x) w(r
to
the solution w(x, 1) t 1. We use the same data of Burgers' equation provided in Li Li al. 202
The initial state wo(x) is generated from the Gaussian random field (( (-(( 257)
with the periodic boundary conditions. The split step and the fine forward Euler methods were employed to generate a solution at f = 1. We set viscosity v and a spatial resolution to 0.1 and 27 128, respectively. The size of the training sample and test sample we used are 1,000 and 200, respectively.
We take the ReLU activation function and the InverseTimeDecaa scheduler to experiment with the same setting as in Lu al. (2022). For fair comparison, all experiments DeepONet retained
21

---

Published as a conference paper at ICLR 2023
the hyperparameter values used in Lu et al. (2022) We use the full batch that the batch size is 1,280,000.
E.5 SHALLOW WATER EQUATION
The shallow water equations are hyperbolic PDEs which describe the free-surface fluid flow prob- lems. They are derived from the compressible Navier-Stokes equations. The physical conservation laws of the mass and the momentum holds in the shock of the solution. The specific form of the equation can be written as
Oh 0 (hu) + 2 (hv) 0,
+
@(hu)
(n² gghh² + (hwv) 0,
öt
8z
(14)
0(hm)
+
By
a
( + ggh²) + 8y (hwr) = 0,
x1, ho (21, (0 x2);
fort E [0, 1] and 21. 22 E (12255 2.5] where h(t, 21, I22) denotes the height of water with horizontal and vertical velocity (u. v). denotes the gravitational acceleration. In this paper, we aim learn the operator ho (2))),022 > (1(1, ; (2)) re/14/41)411 without the information of (u, v). For the sampling of initial conditions and the corresponding solutions, we directly followed the setting of Takamoto jct al. (2022). The 2D radial dam break scenario considered so that the initialization of the water height is generated as circular bump in the center of the domain. The initial condition is generated by
2.0, for V
h(t = 0, x2) =
(15)
1.0, for
with the radius randomly sampled from ((0.3, 0.7]. The spatial domain determined to be a 2- dimensional rectangle [-2.5, 2.5) x 1-5.5,2.5|, We use 256 = =62 grids for the spatial domain. We train the models with three snapshots at 0.25, 0.5, 0.75, and predict the solution hit,, ø1, x22) for four snapshots t 0.25, 0.5, 0.75, 1 on the same grid. We use 100 training samples and the batch size is determined to be 25,600.
F ADDITIONAL EXPERIMENTS
F.1
COMPARISON UNDER VARIOUS CONDITIONS
The experimental results under various conditions are included in Ty modifying the network structure, activation function, and number of sensor points. Although the DeepONet shows good performance in certain settings, the proposed HyperDeepONet shows good performance without dependency on the various conditions.
Activation: ReLU, M = 30
DeepONet
Shift
Flex
NOMAD
Hyper(Ours)
d,-30-30-30-11
0.16797
1.30852
1.04292
0.27209
0.02059
Target network
d.-50-550-1
0.04822
1.08760
1.11957
0.21391
0.05562
Activation: ReLU, MM 100
DeepONet
Shift
Flex
NOMAD
Hyper(Ours)
Target network d,-30-33-30-1
0.02234
1.08310
1.03741
0.19089
0.01743
4,-500-50-1
0.07255
1.47373
1.13217
0.14020
0.04645
Activation: PReLU, M 30
DeepONet
Shift
Flex
NOMAD
Hyper(Ours)
d,-30-30-30-1
0.11354
1.09395
1.03502
0.25651
0.02844
Target network
d.-550-50-1
0.00873
1. 1.14073
1.06947
0.04054
0.04302
Activation: PReLU, M = 100
DeepONet
Shift
Flex
NOMAD
Hyper(Ours)
dy-30-30-30-1
0.01035
1.05080
1.07791
0.16592
0.01083
Target network
-0-50--0--1
0.07255
1.47373
1.13217
0.14020
0.04645
Table 5: The relative 2 test errors for experiments on training the identity operator under various conditions
22

---

Published as a conference paper at ICLR 2023
0.5
0.5
0.4
0.4
Rel L2 error
Rel L2 error
train_ DeepONet
train, DeepONet
0.3
test DeepONet
0.3
test DeepONet
train HyporDeepONet
tro n, HypcrDeepONict
test HyperDrepONet
HyperDnepONet
0.2
0.2
0.1
0.1
0.0
1
5
3
# hidden layers in branch net (hypernetwork) # hidden layers in branch net (hypernetwrrk)
Figure 11: Varying the number of layers of branch net and hypernetwork in DeepONet and Hyper- DeepONet for identity operator problem (left) and differentiation operator problem (right).
F.2 VARYING THE NUMBER OF LAYERS IN BRANCH NET AND HYPERNETWORK
Figure compares the relative 2 error of the training data and test data for the DeepONet and the HyperDeepONett by varying the number of layers in the branch net and the hypernetwokk while maintaining the same small target network. Note that the bottom 150 training and test data with lower errors are selected to observe trends cleary. The training and test error for the DeepONet is not reduced despite the depth of the branch net becoming larger. This a limitation of DeepONet's linear approximation. DeepONet approximates the operator with the dot product of the trunk net's output that approximates the basis of the target function and the branch net's output that approxi- mates the target function's 's coefficient. Even if a more accurate coefficient is predicted by increasing the depth of the branch net, the error does not decrease because there is a limit to approximating the operator with linear approximation using the already fixed trunk net.
The NepeoNe approximates the operator with a low test error in all cases with a different number of layers. Figure 11 shows that the training error of the HyperDeeeONeet remains small as the depth of the hypernetwork increases, while the test error increases. The increasing gap between the training and test errors is because of overfitting. HyperDeepONet overfits the training data because the learnable parameters of the model are more than necessary to approximate the target operator.
F.3 COMPARISON OF HYPERDEEPONET WITH FOURIER NEURAL OPERATOR
The Fourier Neural Operator (FNO) Li et al. 2021) is a well-known method for operator learning. Lu et al. (2022) consider 16 different tasks to explain the relative performance of the DeepONet and the FNO. They show that each method has its advantages and limitations. In particular, DeepONet has great advantage over FNO when the input function domain is complicated, or the position of the sensor points is not uniform. Moreover, the DeepONet and the HyperDeepONet enable the inference of the solution of time-dependent PDE even in finer time grid than time grid used for training. t.e continuous-ing in-time solution operator of the shallow water equation in our experi- ment. Since the FNO is image-to- to-image based operator learning model, it cannot obtain a continuous solution operator over time / and position æ1, .22. In this paper, while retaining these advantages of DeepONets, we focused on overcoming the difficulties of DeepONets learning complex target func- tions because of linear approximation. Therefore, mainly compared the vanilla DeepONet and its variants models to learn the complex target function without the result of the FNO.
Table the simple comparison of the HyperDeepONet with the FNO for the identity operator and differentiation operator problems. Although the FNO structure has four Fourier layers, we use only one Fourier layer with 2,4,8, and 16 modes for fair comparison using similar number of parameters. The FNO shows a better performance than the HyperDeepONet for the identity operator problem. Because the FNO has linear transform structure with Fourier layer, the identity
23

---

Published as a conference paper at ICLR 2023
Fourier Neural Operator
Model
HyperDeepONeet (ours)
Mode 2
Mode 4
Mode 8
Mode 16
Identity
0.0358
0.0005
0.0004
0.0003
0.0004
Differentiation
0.1268
0.8256
0.6084
0.3437
0.0118
Param
(or4l(or 16741)
20993
29185
45569
78337
Table 6: The relative L test errors and the number of parameters for the identity and differentia- tion operator problems using HeperDeepONet and FNO with different number of modes. #Paaam denote the number of learnable parameters.
operator is easily approximated even with the 2 modes. In contrast, the differentiation operator is hard to approximate using the FNO with 2, 4, and 8 modes. Although the FNO with mode 16 can approximate the differentiation operator with better performance than the HyperDeepONet, requires approximately 4.7 times as many parameters as the HyperDeepONet..
Model
DeepONet
Shift
Flex
NOMAD
Hyper(Ours)
Param
274K
281K
282K
270K
268K
Advection
Rel error
0.0046
0.0095
0.0391
0.0083
0.0048
Param
115K
122K
122K
117K
114K
Burgers
Rel error
0.0391
0.1570
0.1277
0.0160
0.0196
# Param
107K
111K
117K
TOIK
Shallow
Rel error
0.0279
0.0299
0.0167
0,0148
Shallow
#aara 6.5K
8.5K
6.4K
5.6K
w/ small param
Rel error
0.0391
0.0380
0.0216
0,0209
Table 7: The relative L test errors and the number of parameters for the solution operators of PDEs experiments. #Param denote the number of learnable parameters. Note that the all five models use the similar number of parameters for each problem.
F.4 PERFORMANCE OF OTHER BASELINES WITH THE SAME NUMBER OF LEARNABLE PARAMETERS
For three different PDEs with complicated target functions, compare all the baseline methods in Table evaluate the performances. We analyze the model's computation efficiency based on the number of parameters and fix the model's complexity for each equation. All five models demon- strated their prediction abilities for the advection equation. DeepONet shows the greatest perfor- mance in this case, and other variants equation, NOMAD and HeperDeepPNet are the two outstanding algorithms from the perspective of relative test error. NOMAD seems slightly dominant our architectures, but the two models compete within the margin of error. Furthermore, HyperDeepONet improves its accuracy using the chunk embedding method, which enlarge the target network's size while maintaining the complex- ity. Finally, HyperDeepONet and NOMAD outperform the other models for 2-dimensional shallow water equations. The HyperDeepONet still succeeds in accurate prediction even with few pa- rameters. It can be observed from Table that NOMAD is slightly more sensitive to extreme case using a low-complexity model. Because of the limitation in computing 3-dimensional rotation, FlexDeepONet cannot be applied this problem.
longer improve the performance. For the Burgers'
Figure 13 shows the results on prediction of shallow water equations' solution operator using the DeepONet and the HyperDeepONet. The overall performance of the DeepONet is inferior to that of the HyperDeepONNt, which is consistent with the result in Figure 12 particular, the DeepONet has difficulty matching the overall circular shape of the solution when the number of parameters is small. This demonstrates the ad advantages the HyperDeepONet when the computational resource limited.
24

---

Published as a conference paper at ICLR 2023
(a) Shallow water equations with large models
(b) Shallow water equations with small models
Test Relative Eiror
Test Relltivee Error
8x
C.A.
* 4x100
nomad
3x20
1x
hyper
1x100
2x100
5000
7500
1,000
12500
15000
17500
20000
2300
5000
10000
32500
15000
17500
20000
Epoch(s)
Epoch(s)
Figure 12: The test L relative enrors of four methods during training for the solution operator of shallow water equations.
Training time (8)
Inference time (ms)
(per epoch)
DeepONet
1.018
0.883
Same target (Differentiation)
HeperDeepONet 1.097
1.389
DeepONet
0.466
0.921
Same #param (Advection)
HyperDeepONee
0.500
1.912
Table 8: The training time and inference time for the differentiation operator problem and the solu- tion operator of advection equation problem using DeepONet and HyperDeeppNNet.
F.5 COMPARISON OF TRAINING TIME AND INFERENCE TIME
Table g shows the training time and the inference time for the DeepONet and the Oeper for two different operator problems. When the same small target network is employed for the Deep- ONet and the HyperDeepONet, the training time and inference time for the HyperDeepONet are larger than for the DeepONet. However, in this case, the time is meaningless because DeepONet does not learn the operator with the desired accuracy at all (Table and Figure (
Even when both models use the number of training parameters, HyperDDepPONet takes slightly longer to train for one epoch than the DeepONet. However, the training complex operator using the HyperDeepONet takes fewer epochs to get the desired accuracy than DeepONet, seen in Figure 7 This phenomenon can also be observed for the shallow water problem in Figure 12] It shows that the HyperDeepONet converges the desired accuracy faster than any other variants of DeepONet.
The HyperDeepONet also requires larger inference time because it can infer the target network after the hypernetwork is used generate the target network's parameters. However, when the input function's 's sensor values are already fixed, the inference time to predict the output of the target function for various query points faster than that of the DeepONet. This because the size of the target network for HyperDeepONet is smaller than that of the DeepONet, although the total number of parameters is the same.
25

---

Published as a conference paper at ICLR 2023
DeepONet
HyperDespONet
DeepONet
Ismall)
1.8
1.4
1.0
ONe (small)
DeepONel (smail)
Exoct So Luttion
HyperDaspONet
DaepONet
1.4
1.2
1.0
AL
HyperDeeppOeet (smaill
DeepONet (smalll
Solution
HyperDeepONet
DeepONet
1.8
1.1
HyperDeapONet
DeepONet
DeperOeepONeet
133
11
Figure 13: The examples of predictions on the solution operator of shallow water equations using the DeepONet and the HyperDeepONet. The first column represents the exact solution generated in Takamoto al. (2022), and the other four columns denote the predicted solutions using the corresponding methods. The four rows shows the predictions h(t, 21, F2) at four snapshots = [0.25, 0.5, 0.75, 1].
26


Reviews:
[{"review_id": "X54OOcDPbg", "reviewer": "Reviewer_eaJx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an extension of the deep operator network (DeepONet), originally proposed in (Lu et al., 2019). The DeepONet is a way to learn operators, i.e., mappings between two functions. It takes a set of values from the first function (called sensors) and a query from the second function in input, and returns the value of the second function evaluated on the query point.\n\nThe original DeepONet framed this as learning an inner product between the representation of the sensors (branch) and the representation of the query. This paper instead proposes to parameterize directly a second neural network with the branch network, inspired by hypernetworks. They showcase on two experiments that this allows to reduce the complexity of the network. They also provide a number of theoretical insights on the reason.", "summary_of_the_review": "My suggestion from reading this paper and the original DeepONet paper is that the contribution appears to be valuable, although the paper is not easy to follow and the experimental part appears weak if compared to the DeepONet paper. However, for the reasons mentioned above, this is only a partial evaluation and I cannot trust there are similar papers in the state-of-the-art, or the theoretical derivation has points that I was not able to evaluate carefully.", "strength_and_weaknesses": "I premise this review by saying I am not an expert on the field of operator learning, and I had to read the DeepONet paper to provide some judgment. In particular, I was not able to follow the theoretical derivations in Section 4 completely.\n\nThe setup reminds me of [1], where they show a Transformer network can do Bayesian inference; in both cases, the network at test time is provided with a dataset and a query point and learns to do inference by exploiting a transformer.\n\nFrom what I understand, the connection made here with the hypernetwork is interesting and the paper does provide a strong theoretical / empirical motivation for replacing the original DeepONet with this variant. On the experimental part, I think more details on the actual architectures (apart from saying \"fully-connected networks\") should be provided.\n\nThe paper is not particularly easy to follow without reading (Lu et al., 2019) first. For example, \"sensor values\" are mentioned multiple times but only defined in the middle of the paper. Similarly, \"p-coefficients and p-basis\" is unclear in the introduction. In general, the motivation for using this architecture as compared to, e.g., simply concatenating the sensors values and the query point is not provided.\n\n[1] https://arxiv.org/pdf/2112.10510.pdf ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U2iM_uZt7R", "reviewer": "Reviewer_LzY8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets a cutting-edge research problem that tries to use deep neural networks to solve partial differential equations (PDEs) through operator learning, with the potential to achieve faster and/or more accurate predictions for complex physical dynamics than traditional numerical methods. Although previously proposed methods (e.g., DeepONet and its variants) achieve some success, due to the limitation of linear approximation investigated by recent work, they typically need a large number of parameters and computational costs to learn the complex operators. In order to solve the problem, the authors propose HyperDeepONet, which learns complex operators with a target network whose parameters are generated by a hypernetwork (conditioned on the input function $u(\\cdot)$). Both theoretical and empirical evidence is provided to demonstrate the lower complexity and effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper focuses on a significant research problem and proposes an interesting method to make learning complex operators with lower complexity (e.g., fewer parameters) possible. Well-rounded theory and empirical results are provided. I believe this work could be further improved with better writing and further evaluation. Based on the current version, I recommend a borderline acceptance.\n\n**Update on 04 Nov after reading two other reviews:** I agree with the other two reviewers, and my main concerns about this work are: a) the writing is not good (*eaJx*); b) the evaluation is not strong (*8Dyr*). If the authors could resolve these two concerns and update the draft accordingly before 18 Nov, I would be happy to recommend this work.", "strength_and_weaknesses": "**Pros:**\n\n* The motivation is clear, and the idea of leveraging a hypernetwork to generate parameters of the target network is interesting.\n* The authors analyze DeepONet with its variants from the perspective of: \"how information from the input function $u$ is injected into the target network $\\mathcal{G}_{\\theta}(u)$\" as illustrated in Figure 2. It is quite clear and makes connections among these methods, including HyperDeepONet.\n* Thorough theoretical analysis of the complexity is provided (but I cannot follow this part).\n\n**Cons:**\n\n* The writing of this paper could be further polished. Many typos also exist.\n\n* About the experiment section: I do not see the results of FNO [1], which also belongs to operator learning. Could the authors explain the reason? Moreover, A PDE benchmark [2] has been proposed recently. It would be better to provide some experimental results on it (the second half does not affect my evaluation since it was proposed recently).\n* The scalability of the proposed method seems to be an issue. This paper mainly considers the complexity of the target network. It would be better to consider the complexity of the branch network (also the hypernet) as well in the proposed method (i.e., the complexity of the Hyper will increase as that of the target network increases). \n\n**Question:**\n\n* About the experiment section, \n  * in Figure 7 and 8, could the authors explain why increasing the number of layers in the branch net does not reduce the $L^2$ error of DeepONet. For the HyperDeepONet, why it may even increase the error? What is the intuition behind this phenomenon?\n  * when considering the whole complexity (i.e., the same total Params in table 2), the performance of Hyper seems to be inferior to that of DeepONet. How about the performance of other baselines? \n* The authors mention that real-time prediction on resource-constrained hardware is crucial and challenging, so for each method, how about the training time, and inference time to obtain a desired prediction?\n\n**Minor:**\n\n* Please explicitly define the abbreviation when they first appear, e.g., PDEs on Page 1.\n\n* \"shows\" -> \"show\"; \"small\" -> \"a small\" in point 3 of main contributions.\n* \"solving\" -> \"solve\"; \"nonlinear\" -> \"a nonlinear\" in the second paragraph of related work.\n* \"considered\" -> \"considered as\" in section 3.3.\n\n[1] Fourier neural operator for parametric partial differential equations.\n\n[2] PDEBench: An Extensive Benchmark for Scientific Machine Learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LpbSMLn1tWI", "reviewer": "Reviewer_8Dyr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper builds on the deep operator network (DeepONet) for learning operators between function spaces, for examples for PDE solving. This paper proposes HyperDeepONet which replaces the network with a hypernetwork, in other words making the target function input-dependent. This method is shown theoretically to be more parameter efficient and improves on related baselines on synthetic experiments.\n", "summary_of_the_review": "This paper provides an improvement to the deep operator network approach to operator learning, which is well motivated and has promising empirical results.\n\n----------\nPost rebuttal:\nThe authors have provided new content, figures, and experiments addressing my concerns and overall improved the presentation of the paper. I am increasing my score. ", "strength_and_weaknesses": "Strengths:\n- Well-explained problem setup and background\n- Method is well motivated\n- Method comes with theoretical results for both upper and lower bounds (Theorem 1 and 2)\n- Well-rounded empirical study including metrics, visualization of predictions, and several baselines and ablations\n\nWeaknesses:\n- Evaluation is very small-scale, consisting of largely of synthetic data and simple PDEs, and is hard to tell whether the proposed method can be of practical significance on more interesting problems. While results look promising, there are ultimately very few quantitative results and it is hard to evaluate the efficacy of the method.\n- A discussion of the tradeoffs of the proposed method is missing (i.e. potential weaknesses). For example, when the network is more complicated as in HyperDeepONet, how is the computational speed compared to a parameter-matched DeepONet?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster