SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computing this derivative requires feature representations  $f_{j}^{(\theta_{i})}$ , which can be reused from the forward pass results computed beforehand. The entire  $r$  components of the feature derivatives  $\partial f^{(\theta_i)}(x_i) / \partial x_i:\mathbb{R}\to \mathbb{R}^r$  can be obtained by a single pass thanks to forward-mode AD. To obtain the full gradient  $\nabla \hat{u} (x):\mathbb{R}^d\to \mathbb{R}^d$ , we iterate the calculation of Eq. 7 over  $d$  times, switching the input axis  $i$ . Also, since each iteration involves  $N$  training samples, the total number of JVP evaluations becomes  $Nd$ , which is consistent with Eq. 3. Note that any  $p$ -th order derivative  $\partial^p\hat{u} /\partial x_i^p$  can also be obtained in the same way.
(a) Diffusion
(b) Helmholtz
(c)  $(2 + 1)$  -d Klein-Gordon
Figure 6: Overall results of (a) diffusion, (b) Helmholtz, (c)  $(2 + 1)$ -d Klein-Gordon, and (d)  $(3 + 1)$ -d Klein-Gordon experiments. It shows comparisons among PINN, PINN with modified MLP (PINN-mod), SPINN, and SPINN with modified MLP (SPINN-mod). For each experiment, the first to third columns show the relative error, runtime, and GPU memory versus different numbers of collocation points, respectively. The rightmost column shows the training curves of each model when the number of collocation points is  $64^{3}$  ( $54^{3}$  for PINN with modified MLP.) For  $(3 + 1)$ -d Klein-Gordon, the training curve is plotted when each model is trained with the number of collocation points of  $16^{4}$ . The error and the training curves are averaged over 7 different runs and  $70\%$  confidence intervals are provided. Note that the y-axis scale of every plot is a log scale.
(d)  $(3 + 1)$ -d Klein-Gordon
4.4 Universal Approximation Property
It is widely known that neural networks with sufficiently many hidden units have the expressive power of approximating any continuous functions [6, 18]. However, it is not straightforward that our suggested architecture enjoys the same capability. For the completeness of the paper, we provide a universal approximation property of the proposed method.
Theorem 1. (Proof in appendix) Let  $X, Y$  be compact subsets of  $\mathbb{R}^d$ . Choose  $u \in L^2(X \times Y)$ . Then, for arbitrary  $\varepsilon > 0$ , we can find a sufficiently large  $r > 0$  and neural networks  $f_j$  and  $g_j$  such that
$$
\left\| u - \sum_ {j = 1} ^ {r} f _ {j} g _ {j} \right\| _ {L ^ {2} (X \times Y)} <   \varepsilon . \tag {8}
$$
By repeatedly applying Theorem 1, we can show that SPINN can approximate any functions in  $L^2$  in the high-dimensional input space. An approximation property for a broader function space is a fruitful research area, and we leave it to future work. To support the expressive power of the suggested method, we empirically showed that SPINN can accurately approximate solution functions of various challenging PDEs, including diffusion, Helmholtz, Klein-Gordon, and Navier-Stokes equations.
We compared SPINN against vanilla PINN [34] on 3-d (diffusion, Helmholtz, Klein-Gordon, and Navier-Stokes equation) and 4-d (Klein-Gordon and Navier-Stokes) PDE systems. Every experiment was run on a different number of collocation points, and we also applied the modified MLP introduced in [44] to both PINN and SPINN. For 3-d systems, the number of collocation points of  $64^{3}$  was the upper limit for the vanilla PINN ( $54^{3}$  for modified MLP) when we trained with a single NVIDIA RTX3090 GPU with 24GB of memory. However, the memory usage of our model was significantly smaller, enabling SPINN to use a larger number of collocation points (up to  $256^{3}$ ) to get more accurate solutions. This is because SPINN stores a much smaller batch of tensors, which are the primals ( $v_{k}$  in Tab. 1) while building the computational graph. All reported error metrics are average relative  $L_{2}$  errors computed by  $\| \hat{u} - u\|^2 / \| u\|^2$ , where  $\hat{u}$  is the model prediction, and  $u$  is the reference solution. Every experiment is performed seven times (three times for  $(2 + 1)$ -d and five times for  $(3 + 1)$ -d Navier-Stokes, respectively) with different random seeds. More detailed experimental settings are provided in the appendix.
Fig. 6 shows the overall results of forward problems on three 3-d systems (diffusion, Helmholtz, and Klein-Gordon) and one 4-d system (Klein-Gordon). SPINN is significantly more computationally efficient than the baseline PINN in wall-clock run-time. For every PDE, SPINN with the modified MLP found the most accurate solution. Furthermore, when the number of collocation points grows exponentially, the memory usage and the actual run-time of SPINN increase almost linearly. We also confirmed that we can get more accurate solutions with more collocation points. This training characteristic of SPINN substantiates that our method is very effective for solving multi-dimensional PDEs. Furthermore, with the help of the method outlined in Griewank and Walther [13], we estimated the FLOPs for evaluating the derivatives. Compared to the baseline, SPINN requires  $1,394 \times$  fewer operations to compute the forward pass, first and second order derivatives. Further details regarding the FLOPs estimation are provided in section C of the appendix. In the following sections, we give more detailed descriptions and analyses of each experiment.
Diffusion Equation When trained with  $128^{3}$  collocation points, SPINN with the modified MLP finds the most accurate solution. Furthermore, when trained with the same number of collocation points  $(64^{3})$ , SPINN-mod is  $52\times$  faster and  $29\times$  more memory efficient than the baseline with modified MLP. You can find the visualized solution in the appendix. Since we construct the solution relatively simple compared to other PDE experiments, baseline PINNs also find a comparatively good solution. However, our model finds a more accurate solution with a larger number of collocation points with minor computational overhead. The exact numerical values are provided in the appendix.
Helmholtz Equation The results of the Helmholtz experiments are shown in Fig. 6b. Due to stiffness in the gradient flow, conventional PINNs hinder finding accurate solutions. Therefore, a modified MLP and learning rate annealing algorithm is suggested to mitigate such phenomenon [44]. We found that even PINNs with modified MLP fail to solve the equation when the solution is complex (contains high-frequency components). However, SPINN obtains one order of magnitude lower relative error without bells and whistles. This is because each body network of the SPINN learns an individual 1-dimensional function which mitigates the burden of representing the entire 3-dimensional function. This makes our model much easier to learn complex functions. Given the same number of collocation points  $(64^{3})$ , the training speed of our proposed model with modified MLP is  $62\times$  faster, and the memory usage is  $29\times$  smaller than the baseline with modified MLP. The exact numerical values are provided in the appendix.
Klein-Gordon Equation Fig. 6c shows the results. Again, our method shows the best performance in terms of accuracy, runtime  $(62\times)$ , and memory usage  $(29\times)$ . Note that both Helmholtz and Klein-Gordon equations contain three second-order derivative terms, while the diffusion equation contains only two. Our results showed the largest differences in runtime and memory usage in Helmholtz and Klein-Gordon equations. This is because SPINN significantly reduces the AD computations with forward-mode AD, implying SPINN is very efficient for solving high-order PDEs.
We investigated the Klein-Gordon experiment further, extending it to a 4-d system by adding one more spatial axis. As shown in Fig. 6d, baseline PINN can only process  $23^{4}$  ( $18^{4}$  for modified MLP) collocation points at once due to a high memory throughput. On the other hand, SPINN can exploit the number of collocation points of  $64^{4}$  ( $160 \times$  larger than the baseline) to obtain a more precise solution. The exact numerical values are provided in the appendix.
(2+1)-d Navier-Stokes Equation We constructed SPINN to predict the velocity field  $u$  and applied forward-mode AD to obtain the vorticity  $\omega$ . We used the same PDE setting used in causal PINN [43] and compared SPINN against their model. As shown in Tab. 2, our model finds a comparably accurate solution even without the causal loss function. Furthermore, since causal PINN had to iterate over four different tolerance values  $\epsilon$  in the causal loss function, they had to run  $3\sim 4$  times more training epochs, depending on the stopping criterion. As a result, given the same number of collocation points, SPINN converges  $60\times$  faster than causal PINN in terms of training runtime. The results of the Navier-Stokes equation ensure that SPINN can successfully solve a chaotic, highly nonlinear PDE. In Fig. 7, we
Figure 7: Relative error and training speed of  $(2 + 1)$ -d Navier-Stokes experiment vs. different ranks of SPINN.
Table 2: The numerical result of  $(2 + 1)$ -d Navier-Stokes equation compared against PINN with modified MLP (PINN+mod) and causal PINN.  $N_{c}$  is the number of collocation points. The fourth row shows the training runtime of a single time window in time-marching training. All relative  $L_{2}$  errors are averaged over 3 runs.
† This is the error (the single run result) reported by the causal PINN paper. Since the causal PINN is sensitive to the choice of parameter initialization, we also reported average error obtained from different random seeds by running their official code.
model	PINN+mod [44]	causal PINN [43]	SPINN (ours)
\( N_c \)	\( 2^{12} \)	\( 2^{15} \)	\( 2^{12} \)	\( 2^{15} \)	\( 2^{15} \)	\( 2^{15} \)	\( 2^{18} \)	\( 2^{21} \)
relative \( L_2 \) error	0.0694 ±0.0091	0.0581 ±0.0135	0.0578 ±0.0117	0.0401 ±0.0084	\( 0.0353^† \)	0.0780 ±0.0049	0.0363 ±0.0018	0.0355 ±0.0006
runtime (hh:mm)	03:20	07:52	10:09	23:03	-	00:07	00:09	00:14
memory (MB)	5,198	17,046	5,200	17,132	-	764	892	1,276
demonstrate the effect of the rank  $r$  for solving the equation. We observed that the performance almost converges at the rank of 128, and increasing the rank does not slow down the training speed too much. Further details and visualization of the predicted vorticity map is shown in appendix section D.4.
(3+1)-d Navier-Stokes Equation We proceeded to explore our model on (3+1)-d Navier-Stokes equation by devising a manufactured solution corresponding to the Taylor-Green vortex [41]. It models a decaying vortex, widely used for testing Navier-Stokes numerical solvers [8]. Since the vorticity of the (3+1)-d Navier-Stokes equation is a 3-d vector as opposed to the (2+1)-d system, the equation has three independent components resulting in a total of 33 derivative terms (see section D.5 in the appendix). Similar to the (2+1)-d experiment, the network's output is the velocity field, and we obtained 3-d vorticity by forward-mode AD. Tab. 3 shows the relative error, runtime, and GPU memory. Trained with a single GPU, SPINN achieves a relative
error of 1.9e-3 less than 30 minutes. Visualized velocity and vorticity vector fields are provided in the appendix.
Table 3: The numerical result of (3+1)-d Navier-Stokes equation.  $N_{c}$  is the number of collocation points on the entire domain. The relative errors are obtained by comparing the vorticity vector field and averaged over five different runs.
model	Nc	relative L2 error	runtime (mins)	memory (MB)
SPINN	84	0.0090	15.33	768
164	0.0041	16.83	1,192
324	0.0019	26.73	2,946
Although we presented comprehensive experimental results to show the effectiveness of our model, there remain questions on solving more challenging and higher dimensional PDEs. Handling any geometric surface is one of the key advantages of PINNs. In the appendix, we presented a simple way to handle arbitrary domains with SPINN by testing on the Poisson equation in an L-shaped domain. Please see section E.3 for more details. Another method could be applying an additional operation after the SPINN's feature merging to map a rectangular mesh to an arbitrary physical mesh, inspired by PhyGeoNet [11] and Geo-FNO [29]. Combining such techniques for handling arbitrary boundary conditions of complex geometries into ours is an exciting research direction, and we leave it to future work.
We also found that due to its architectural characteristic, SPINN tends to be better trained when the solutions align with the variable separation form (discretized into low-rank tensor). However, note that SPINN is still effective in solving equations where the solution is not a low-rank tensor, such as our examples on diffusion and  $(2 + 1)$ -d Navier-Stokes equation. Please see section D in the appendix for additional experiments and information of which example aligns with the variable separation form or not. Applying SPINN to higher dimensional PDEs, such as the BGK equation, would also have a tremendous practical impact. Lastly, we are still far from the theoretical speed-up (wall-clock runtime vs. FLOPs), which is further discussed in appendix section C. We expect to additionally reduce runtime by optimizing hardware/software, e.g., using customized CUDA kernels.
We showed a simple yet powerful method to employ large-scale collocation points for PINNs training, leveraging forward-mode AD with the separated MLPs. Experimental results demonstrated that our method significantly reduces both spatial and computational complexity while achieving better accuracy on various three and four-dimensional PDEs. To our knowledge, it is the first attempt to exploit the power of forward-mode AD in PINNs. We believe this work opens up a new direction to rethink the architectural design of neural networks in many scientific applications.
This research was supported by the Ministry of Science and ICT (MSIT) of Korea, under the National Research Foundation (NRF) grant (2022R1A4A3033571), Institute of Information and Communication Technology Planning Evaluation (IITP) grants for the AI Graduate School program (IITP-2019-0-00421), Korea Institute of Energy Technology Evaluation and Planning (KETEP) and the Ministry of Trade, Industry & Energy (MOTIE) of the Republic of Korea (No. 20224000000360).
[1] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699-4711, 2021.
[2] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, pages 1-43, 2018.
[3] John P Boyd. Chebyshev and Fourier spectral methods. Courier Corporation, 2001.
[4] Steven L Brunton and J Nathan Kutz. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press, 2022.
[5] Pao-Hsiung Chiu, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, and Yew-Soon Ong. Can-pinn: A fast physics-informed neural network based on coupled-automatic-numerical differentiation method. Computer Methods in Applied Mechanics and Engineering, 395:114909, 2022.
[6] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[7] Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Rethinking the importance of sampling in physics-informed neural networks. arXiv preprint arXiv:2207.02338, 2022.
[8] C Ross Ethier and DA Steinman. Exact fully 3d navier-stokes solutions for benchmarking. International Journal for Numerical Methods in Fluids, 19(5):369-375, 1994.
[9] Robert Eymard, Thierry Gallouet, and Raphaele Herbin. Finite volume methods: handbook of numerical analysis. PG Ciarlet and JL Lions (Eds), 2000.
[10] Charles L Fefferman. Existence and smoothness of the navier-stokes equation. The millennium prize problems, 57:67, 2000.
[11] Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428:110079, 2021.
[12] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
[13] Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
[14] Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021.
[36] Shyam Sankaran, Hanwen Wang, Leonardo Ferreira Guilhoto, and Paris Perdikaris. On the impact of larger batch size in the training of physics informed neural networks. In The Symbiosis of Deep Learning and Differential Equations II, 2022.
[37] Ramansh Sharma and Varun Shankar. Accelerated training of physics informed neural networks (pinns) using meshless discretizations. Advances in Neural Information Processing Systems, 2022.
[38] Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.
[39] Gordon Dennis Smith. Numerical solution of partial differential equations: finite difference methods. Oxford university press, 1985.
[40] Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
[41] Geoffrey Ingram Taylor and Albert Edward Green. Mechanism of the production of small eddies from large ones. Proceedings of the Royal Society of London. Series A-Mathematical and Physical Sciences, 158(895):499-521, 1937.
[42] Nils Wandel, Michael Weinmann, Michael Neidlin, and Reinhard Klein. Spline-pinn: Approaching pdes without data using fast, physics-informed hermite-spline cnns. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8529-8538, 2022.
[43] Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022.
[44] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055-A3081, 2021.
[45] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. In Computer Graphics Forum, pages 641-676. Wiley Online Library, 2022.
[46] Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81, 2019.
[47] Olek C Zienkiewicz, Robert Leroy Taylor, and Jian Z Zhu. The finite element method: its basis and fundamentals. Elsevier, 2005.
Separable PINN - Supplementary Materials
Here we show the preliminary lemmas and proofs for Theorem 1. in the main paper. We start by defining a general tensor product between two Hilbert spaces.
Definition 1. Let  $\{v_{\beta}\}$  be an orthonormal basis for  $\mathcal{H}_2$ . Tensor product between Hilbert spaces  $\mathcal{H}_1$  and  $\mathcal{H}_2$ , denoted by  $\mathcal{H}_1 \otimes \mathcal{H}_2$ , is a set of all antilinear mappings  $A: \mathcal{H}_2 \to \mathcal{H}_1$  such that  $\sum_{\beta} \|Av_{\beta}\|^2 < \infty$  for every orthonormal basis for  $\mathcal{H}_2$ .
Then by Theorem 7.12 in Folland [4],  $\mathcal{H}_1\otimes \mathcal{H}_2$  is also a Hilbert space with respect to norm  $\| \cdot \|$  and associated inner product  $\langle \cdot ,\cdot \rangle$  ..
$$
\left\| A \right\| ^ {2} \equiv \sum_ {\beta} \left\| A v _ {\beta} \right\| ^ {2}, \tag {1}
$$
$$
\langle A, B \rangle \equiv \sum_ {\beta} \left\langle A v _ {\beta}, B v _ {\beta} \right\rangle , \tag {2}
$$
where  $A, B \in \mathcal{H}_1 \otimes \mathcal{H}_2$ , and  $\{v_{\beta}\}$  is any orthonormal basis of  $\mathcal{H}_2$ .
Lemma 1. Let  $x\in \mathcal{H}_1$  and  $y,y^{\prime}\in \mathcal{H}_{2}$  . Then,  $(x\otimes y)y^{\prime} = \langle y,y^{\prime}\rangle x$
Proof. Let  $A$  be a mapping  $A: y' \to \langle y, y' \rangle x$ , where  $\| y \| = 1$ . We expand  $y'$  with orthonormal basis  $\{y, z_{\beta}\}$ . i.e.,  $\{y, z_{\beta}\}$  is a basis for  $\mathcal{H}_2$ . Then,
$$
\begin{array}{l} \left\| A y \right\| ^ {2} + \sum_ {\beta} \left\| A z _ {\beta} \right\| ^ {2} = \left\| A y \right\| ^ {2} + \sum_ {\beta} \left\| \langle y, z _ {\beta} \rangle x \right\| ^ {2} (3) \\ = \| A y \| ^ {2} <   \infty (4) \\ \end{array}
$$
It is obvious that  $A$  is antilinear. Then by Definition 1,  $A \in \mathcal{H}_1 \otimes \mathcal{H}_2$  which is,  $Ay' = (x \otimes y)y'$ . Therefore,  $(x \otimes y)y' = \langle y, y' \rangle x$ .
Lemma 2.  $\{u_{\alpha}\otimes v_{\beta}\}$  is an orthonormal basis for  $\mathcal{H}_1\otimes \mathcal{H}_2$
Proof. Let  $A, B \in \mathcal{H}_1 \otimes \mathcal{H}_2$ , where  $B = u_{\alpha} \otimes v_{\beta}$ . Then by the definition of inner product in Eq. 2,
$$
\begin{array}{l} \langle A, B \rangle = \sum_ {i} \left\langle A v _ {i}, B v _ {i} \right\rangle (5) \\ = \sum_ {i} \left\langle A v _ {i}, \left(u _ {\alpha} \otimes v _ {\beta}\right) v _ {i} \right\rangle (6) \\ = \sum_ {i} \left\langle A v _ {i}, \left\langle v _ {\beta}, v _ {i} \right\rangle u _ {\alpha} \right\rangle \quad (\because \text {L e m m a 1}) (7) \\ = \left\langle A v _ {1}, \left\langle v _ {\beta}, v _ {1} \right\rangle u _ {\alpha} \right\rangle + \left\langle A v _ {2}, \left\langle v _ {\beta}, v _ {2} \right\rangle u _ {\alpha} \right\rangle + \dots + \left\langle A v _ {\beta}, \left\langle v _ {\beta}, v _ {\beta} \right\rangle u _ {\alpha} \right\rangle + \dots (8) \\ = \left\langle A v _ {\beta}, u _ {\alpha} \right\rangle (9) \\ \end{array}
$$
Now we check the Parseval identity:
$$
\begin{array}{l} \sum_ {\alpha , \beta} | \langle A, u _ {\alpha} \otimes v _ {\beta} \rangle | ^ {2} = \sum_ {\alpha , \beta} | \langle A v _ {\beta}, u _ {\alpha} \rangle | ^ {2} \quad (\because u _ {\alpha} \otimes v _ {\beta} = B) (10) \\ = \sum_ {\beta} \| A v _ {\beta} \| ^ {2} (11) \\ = \left\| A \right\| ^ {2}. (12) \\ \end{array}
$$
$\therefore \{u_{\alpha}\otimes v_{\beta}\}$  is a basis.
Now we begin the proof of Theorem 1. in the main paper.
Theorem 1. Let  $X, Y$  be compact subsets of  $\mathbb{R}^d$ . Choose  $u \in L^2(X \times Y)$ . Then, for arbitrary  $\varepsilon > 0$ , we can find a sufficiently large  $r > 0$  and neural networks  $f_j$  and  $g_j$  such that
$$
\left\| u - \sum_ {j = 1} ^ {r} f _ {j} g _ {j} \right\| _ {L ^ {2} (X \times Y)} <   \varepsilon . \tag {13}
$$
Proof. Let  $\{\phi_i\}$  and  $\{\psi_j\}$  be orthonormal basis for  $L^2(X)$  and  $L^2(Y)$  respectively. Then  $\{\phi_i\psi_j\}$  forms an orthonormal basis for  $L^2(X \times Y)$  (Lemma 2). Therefore, we can find a sufficiently large  $r$  such that
$$
\left\| u - \sum_ {i, j} ^ {r} a _ {i j} \phi_ {i} \psi_ {j} \right\| _ {L ^ {2} (X \times Y)} <   \frac {\varepsilon}{2}, \tag {14}
$$
where  $a_{ij}$  denotes
$$
a _ {i j} = \int_ {X \times Y} u (x, y) \phi_ {i} (x) \psi_ {j} (y) d x d y.
$$
On the other hand, by the universal approximation theorem [2], we can find neural networks  $f_{j}$  and  $g_{j}$  such that
$$
\left\| \phi_ {i} - f _ {j} \right\| _ {L ^ {2} (X)} \leq \frac {\varepsilon}{3 ^ {j} \| u \| _ {L ^ {2} (X \times Y)}} \text {a n d} \| \psi_ {j} - g _ {j} \| _ {L ^ {2} (Y)} \leq \frac {\varepsilon}{3 ^ {j} \| u \| _ {L ^ {2} (X \times Y)}}. \tag {15}
$$
We first consider the difference between  $u$  and  $\sum_{i,j}^{r} a_{ij} f_i g_j$ :