Interpolating Neural Network-Tensor Decomposition (INN-TD): a scalable and interpretable approach for large-scale physics-based problems
Authors: Jiachen Guo
Xiaoyu Xie
Chanwook Park
Hantao Zhang
Matthew J. Politis
Gino Domel
Wing Kam Liu
Venue: ICML 2025 poster (2025)
Keywords: Scientific Machine Learning; Large-scale simulation; Partial Differential Equations (PDEs); Numerical Analysis; Tensor decomposition
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: Novel neural networks based on interpolation theory from numerical analysis that has faster training/solving speed and better accuracy for modeling high-dimensional large-scale physics-based PDEs

Abstract: Deep learning has been extensively employed as a powerful function approximator for modeling physics-based problems described by partial differential equations (PDEs). Despite their popularity, standard deep learning models often demand prohibitively large computational resources and yield limited accuracy when scaling to large-scale, high-dimensional physical problems. Their black-box nature further hinders their application in industrial problems where interpretability and high precision are critical. To overcome these challenges, this paper introduces Interpolating Neural Network-Tensor Decomposition (INN-TD), a scalable and interpretable framework that has the merits of both machine learning and finite element methods for modeling large-scale physical systems.  By integrating locally supported interpolation functions from finite element into the network architecture, INN-TD achieves a sparse learning structure with enhanced accuracy, faster training/solving speed, and reduced memory footprint. This makes it particularly effective for tackling large-scale high-dimensional parametric PDEs in training, solving, and inverse optimization tasks in physical problems where high precision is required.

Interpolating Neural Network-Tensor Decomposition (INN-TD): a scalable and interpretable approach for large-scale physics-based problems
Jiachen Guo Xiaoyu Xie Chanwook Park Hantao Zhang' Matthew J. Politis'
Gino Domel
Wing Kam Liu 23
Abstract
these models can make predictions much faster than stan- dard numerical simulations. However, deep learning faces challenges in modeling physical problems such as lower accuracy, instability, and the need for a huge amount of high-fidelity training data. As a result, its applications re- main limited for many engineering fields that require high precision, such as aerospace, semiconductor manufacturing, and earthquake engineering. In this paper, we propose Inter- polating Neural Network-Tene which leverages the merits of both machine learning and numerical analysis to tackle large-scale, high-dimensional physical problems governed by partial differential equations (PDEs).
Deep learning has been extensively employed as powerful function approximator for modeling physics-based problems described by partial dif- ferential equations (PDEs). Despite its popular- ity, standard deep learning models often demand prohibitively large computational resources and yield limited accuracy when scaling to large-scale, high-dimensional physical problems. Their black- box nature further hinders the application in in- dustrial problems where interpretability and high precision are critical. To overcome these chal- lenges, this paper introduces Interpolating Neu- ral Network-Tensor Decomposition (INN-TD), scalable and interpretable framework that has the merits of both machine learning and finite element methods for modeling large-scale phys- ical systems. By integrating locally supported interpolation functions from the finite element into the network architecture, INN-TD achieves sparse learning structure with enhanced accu- racy, faster training/solving speed, and reduced memory footprint. This makes it particularly ef- fective for tackling large-scale digh-dimensional parametric PDEs training, solving, and inverse optimization tasks in physical problems where high precision is required.
Decomposition (INN-TD),
2. Related work
Deep learning theory has been applied to solve physical problems that are challenging for standard numerical al- gorithms. Based on the objective of the task, the current works can be broadly classified into major categories: data- driven trainers, data-free solvers, and inverse problems.
2.1. Data-driven trainers
In the data-driven regime, we aim leverage the universal approximation capabilities of neural networks to approxi- mate PDE solutions. Multilayer perceptrons (MLP) have been used due to their universal approximation capability (Cybenko, 1989). Depending on the shape of the domain, convolutional neural networks and graph neural networks can be used as additional inductive biases for data on regu- lar grids and irregular geometries, respectively (Gao et al., 2021; Pfaff et al., 2020). In recent years, many operator learning methods have been proposed to learn functional mappings instead of functions (Li et al., 2020; Lu et al., 2021; Huang et al., 2023). However, due to the black-box nature, designing optimal model architectures and optimiza- tion hyperparameters still largely heuristic and requires trial and error (Rathore et al., 2024; Colbrook et al., 2022). For problems where physics is fully known, it becomes cumbersome to first generate offline data using standard numerical solvers and then train surrogate models from the offline data. Data generation and storage for large-scale
1. Introduction
Deep learning methods have become a popular approach to model physical problems in recent years. Once trained,
'heorretccal and applicd mechanics program, Northwest- ern University, 2145 Sheridan Rd, Evanston, Illinois, USA *Department of mechanical engineering, Northwestern Univer- sity, 2145 Sheridan Rd, Evanston. Illinois, USA 'HIDENN-A,, Evanston, Illinois, USA. Correspondence Wing Kam Liu <w- @ northwestem.ccuu>>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Interpolating Neural Network-Tensor Decompasition (INN-TD)
physical problems can be extremely expensive. For these problems, better approach is to directly solve PDEs using deep learning models.
2.4. Tensor decomposition
Tensor decomposition (TD) has been widely used in data compression tasks (Sidiropoulos al., 2017). Among differ- ent tensor decomposition schemes, the Canonical Polyadic (CP) decomposition has been very popular since it decom- poses higher-order tensors into a tensor product of ID vec- tors (Kolda & Bader, 2009). CP decomposition has also been used vastly to solve PDEs (Beylkin & Mohlenkamp, 2005; Bachmayr, 2023; Dolgov et al., 2012: Cohen et al., 2010). From now on, we refer to CP decomposition as TD unless stated otherwise. TD has also been used in regres- sion and classification tasks as an efficient model due to a lesser number of parameters (Ahmed et al., 2020; Park et al., 2024). Recently, it has been proved that if ML. is used to approximate each mode in TD, then it's also a universal approximation (Vemuri et al., 2025).
2.2. Data-free solvers
For cases where no data is available, deep learning can be used as function approximators to approximate PDE solu- tions. For example, MLP has been vastly used in physics- informed neural networks (PINNs) and its variations to approximate solutions to PDEs (Raissi et al., 2019; Zhang et al., 2022). Despite their vast usage, however, unlike classical numerical solvers which have solid theoretical foundation, most of the current deep learning models are based on heuristics and lack theoretical analysis (Wang et al., 2022). It has been shown that PINN results have often been compared to weak baselines, potentially over- estimating their capabilities (McGreivy& & Hakim, 2024). Most data-free deep learning solvers also suffer from the curse of dimensionality (Krishnapriyan et al., 2021) and are relatively slow compared to standard numerical solvers such as finite differrence/volume/eeennnt methods (Grossmann
3. INN-TD formulation
Interpolating Neural Network- Decomposition (INN- TD) achieves its remarkable efficiency through two key components: the utilization of C-HiDeNN, which is a gen- eral locally supported interpolation theory based on finite element, and the integration of tensor decomposition tech- niques. These elements collectively enhance the frame- work's 's capability to model complex physical systems with improved accuracy and computational efficiency.
et al., 2024).
2.3. Finite element interpolation
Finite element method (FEM) is the computational workhorse to numerically solve differential equations aris- ing in physics- simulation (Liu et al., 2022). FEM is based on finite element (FE) interpolation theory. FEM first decomposes the original computational domain into a finite number of elements and then leverages locally-supported FE interpolation functions to approximate the field variables within each local domain. FEM can solve wide range of problems, including structural mechanics, heat transfer, fluid flow, weather forecasting, and, in general, every con- ceivable problem that can be described by PDEs. Recent advancements in finite element (FE) methods focus on utiliz- ing machine learning to enhance traditional FE interpolation theory. By employing Neural Network (C-HiDeNN), a generalized FE interpola- tion framework has been proposed, offering improved flexi- bility and greater accuracy (Lu et al., 2023; Li et al., 2023; Park et al., 2023). Despite their robustness and widespread application, finite element (FE)-type methods have been primarily applied to address space/space-time problems, exhibiting limitations in handling parametric partial differ- ential equations (PDEs). Furthermore, FE methods are par- ticularly susceptible to the curse of dimensionality, which poses significant computational challenges and escalates costs when applied to large-scale, high-dimensional prob- lems (Bungartz& & Griebel, 2004).
3.1. C-HiDeNN interpolation
Assume we want to learn univariate function f'(x)) given data (Ii, yi). In INN-TD, the target function f(x) is first broken into small elements, and then data within each ele- ment is approximated locally. This concept has been widely used numerical methods such as finite element (FE). Here, we use a generalized version of FEM shape functions called Convolutionall
Deep Neural Network (C-HiDeNN) (Lu et al., 2023) the basis for the local in- terpolation function. For example, to approximate simple sinusoidal function over multiple periods as shown in Fig. 1, we start by dividing the original domain into several el- ements, which we refer to as a mesh. The boundaries of each element are referred to as nodes, which are highlighted as green dots in Fig. 1. For each segment of the mesh, we employ local interpolation functions to perform the ap- proximation locally. The final approximation of the entire function is achieved by linking these locally approximated segments together.
Deep
Assuming approximating the original function using n ele- ments, the univariate function f(x) is approximated using C-HiDeNN interpolation functions in INN-TD, and can be written as:
2

---

Interpolating Neural Network- Decompasition (INN-TD)
In summary, C-HiDeNN basis function has the following advantages over MI.P, especially when applied physical problems: 1. The Dirichlet boundary condition is automat- ically satisfied. 2. C-HiDeNN is interpretable as it uses locally supported basis functions and the number of ele- ments in C-HiDeNN is closely related to the resolution. 3. C-HiDeNN facilitates numerical integration through the use of Gaussian quadrature (Hughes, 2003), making straight- forward and efficient for many classical integral methods in solving PDEs. For simplicity of the notation, hyperparame- ters a, p are dropped from now on.
Original function
INN-TD approximation
10
0.0
-1.0
0.0
0.2
0.4
0.6
0.8
1.0
x
3.2. INN-TD approximation
Figure 1. C-IliDeNN approaximation based on locally supported basis functions
A general multivariate function u( 21E1,,22;.... x2,...,;p)) can be approximated using functional tensor decomposition (TD) as
f(z) R a,p
(1)
M
u(x1, 22, ..... ID) Ju(xx= = Σ (xa)
11 1t
(m)
(6)
where ux E [R is defined as the nodal values (grid point values); N (x; s, p) is the C-HiDeNN interpolation basis functions for different elements as shown in Eq. 2. The approximation is governed by three hyperparameters, which allow for the approximation of polynomial orders of any degree: patch size s, reproducing order p, and dilation pa- rameter a. Details regarding the role of each hyperparameter are provided in the Appendix.
where M defined as the total number of modes in TD; D is the total number of input dimensions; (21, rD): fá p(m) (xaa is the univariate function for d-th dimension and wn-th mode. has been proved that when the mode number M is sufficiently large and each univariate function f (xxa) serves as a universal approxi- mation, Eq. 6 becomes a universal approximation for any multivariate function (Vemuri et al., 2025).
N(x;,, a. ax
= &, (x,3,,,a,)) (2)
In INN-TD, utilize the C-HiDeNN interpolation function for each univariate function få {(m) (xa)). As a result, Eq. can be written as:
C-HiDeNN interpolation function maintains all the essential finite element approximation properties such as Kronecker delta (Hughes, 2003), partition of unity (Melenk & Babuška, 1996), error estimations and convergence theorems (Lu et al., 2023). For example, (1x) satisfies the following Kronecker delta property nodal position IT:
M Ju(x)) = Σ (1 (m)) (m) m==
N2((22) (m).
ND(ED) (m)
The model parameters for each mode in the INN-TD frame- work are ((m) [u (m) .u :.... (m²) Since we have M modes the INN-TD interpolation, the complete model parameters represented U [U7(1) U .... UM)]. The overall structure of INN-TD is shown in Fig. 9 in the Appendix.
(a 8, a, p) = dut
where the Kronecker delta is defined as:
0 if ll
(4)
if=
3.3. Interpretability of INN-TD
At the Dirichlet boundary node (Iz), we have:
As stated in (Doshi-Velez & Kim, 2017), model interpretabil- ity refers to the ability to explain a model's behavior in way that is understandable to humans. INN-TD is an interpretable machine learning algorithm designed for sci-
Σ S. ui
(5)
Thus, unlike many data-free solvers where an additional penalty term has to be added to softly enforce the Dirich- let boundary condition, C-HiDeNN can easily enforce the Dirichlet boundary condition.
For case of discussion, we only cover cases where the outputs are scalars in the main text. Nonetheless, the current framework seamlessly extends handle high-dimensional outputs, as shown in Appendix E.2.6.
3

---

Interpolating Neural Network-Tensor Decompasition (INN-TD)
entific and engineering problems, offering transparency in its structure, clear convergence guarantees, and explain- able insight into how data or parameter changes affect out- comes. In addressing physics- physics-based based problems described using PDEs, we harness several inductive biases inherent in both physics and classical numerical methods. Many physical problems tend to be low-dimensional in nature. For instance, in structural engineering, it is often observed that only the initial few frequencies significantly influence structure's s dynamic response. As such, this response can be effectively approximated using a limited number of modes. More broadly. proper orthogonal decomposition (POD) or Boahhnnen-Lovve expansions are widely employed to sim- plify the complexity inherent in original physical models and have been widely used in reduced-order modeling of structural analysis and fluid dynamics problems (Rathinam utilize INN-TD
shown in (Guo et al., 2025).
y
INN-TD design
100
coarse mesh
5
fine mesh
(60,75)
5
Gaussian
p
(20,25)
s=p=3
$ 3
X
100
& Petzold, 2003). Consequently, an efficient tool to uncover the naturally low-dimensional characteristics of physical problems.
100
Absolute error
1e-6
8.1
80
7.2
6.3
Additionally, INN-TD incorporates the concept of locally supported interpolation functions from the finite element method approximators. Specifically, we employ the C- HiDeNN interpolation function, which balances the robust capabilities of generalized finite element methods with the adaptability of machine learning approaches. As result, INN-TD adeptly accounts for the Kronecker Delta and par- tition of unity properties, providing flcxibility, locality, sta- bility, and control over the interpolation process. Detailed definitions of these desired properties are explained in Ap- pendix A. As result, INN-TI is very efficient and accurate in terms of capturing complex physics for both data-driven training and data-free solving tasks.
5.4
4.5
40
3.6
2.7
0.9
0
0.0
0
20
40
60
80
100
(b)
Figure 2. Interpretable design of INN-TD: the ID meshes used in INN-TD are only refined at the location where nonlinearity happens with larger s and p used to achieve higher-order smooth- ness (b) point-wise absolute error between INN-TD and the exact solution
Finally, unlike many black-box deep learning methods, INN- TD offers a clear interpretation. Specifically, the number of elements ny dimension d determines the resolution of the interpolation, with larger na corresponding to a higher- resolution interpolation in dimension d. For cases where D ==2 2, the total number of modes M is closely linked to the number of singular values in singular value decomposition (SVD) (Kolda & Bader, 2009). The benefits of such inter- pretability when solving PDEs are demonstrated in detail through numerical example in Fig. 2. this example, INN-TD is leveraged to solve the 2D Poisson's equation with the local source term. By controlling mesh density and hyperparameters and p, INN-TD accurately predict the solution field. Details of this example can be found in Appendix B.
3.4. Data-driven training
For training tasks, our goal is to estimate the function y = I Tu(x)) given the data pairs (æ;, y:). In this section, we focus solely on regression tasks, as our primary interest lies in learning function mapping approximate physical relationships' The model parameters U can be obtained via standard data-driven framework. We utilize the mean squared error (MSE) as the loss function to measure the difference between predicted values and true labels:
1 E L K k
U) yt)
Moreover, due to its interpretable nature, INN-TD exhibits convergence properties as data-free solver. Specifically, in- creasing the number of model parameters tends improve accuracy. This is corroborated using multiple numerical ex- amples in Section 4.2. The error bound is also theoretically
where l is the index of training data; K is the total number of training data; (x* , yy is the training data pairs.
We propose two training schemes for determining model
Data-driven training code can be found in: htspp:///tthhi com/ com/ htm/hacchancokkppe / roe/ rev
rev

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
parameters. The first scheme utilizes boosting algorithm, treating each mode weak learner. We iteratively intro- duce new weak learner J (x))) U) until the loss meets the predefined criteria:
Similar FEM, INN-TD employs the integral form, which benefits from the efficiency of Gaussian quadrature-based numerical integration:
J 2pp) Tp, T.) da, dr 0
argmin C argmin U(M)
1 K
Σ (S [(m] w(x )+
(12)
USMy
where 8J iJu((r, Fp Te) is called as the test function (Hughes, 2003). Different test functions can be adopted depending on the formulation. In this paper, we use the Galerkin formulation where the same function space is used for trial and test functions (Hughes, 2003). Additionally, we leverage integration by parts in Eq. 12 to relax the re- quirement for function continuity in INN-TD approximation (Guo et al., 2024).
ulz;; UlM0) - y1
In the second scheme, known as all-at-once optimization, model parameters for all modes are determined simultane- ously.
1 K k
argmin L C= = argmin U
- . (10)
In solving high-dimensional S-P-T partial differential equa- tions (PDEs), a major challenge is the exponential increase in computational cost as the number of dimensions rises. This phenomenon, known as the curse of dimensionality, has been one of the most critical bottlenecks for both standard numerical and deep learning- -bsed solvers. To address this challenge, only one-dimensional integration is performed in Eq. 12 due to the tensor decomposition structure of the INN-TD approximation. This approach ensures that the number of unknowns in the resulting system of equations depends only on the grid points across each dimension. Consequently, INN-TD effectively circumvents the curse of dimensionality, allowing it handle high-dimensional prob- lems using very fine grids and achieving high resolution. This capability is especially advantageous for modeling physical phenomena that demand an extremely fine mesh to accurately capture and resolve localized features.
Details about the boosting and all-at-once training algo- rithms can be found in Appendix C.
3.5. Data-free solving
Contrary to data-driven training tasks, no data available to train the model parameters for data-free solving. How- ever, we know the underlying physics is governed by the corresponding PDE. This knowledge can be leveraged to solve the model parameters in an a priori sense so that the PDE and its initial and boundary conditions are satisfied. As a result, data-free solving tasks are significantly more challenging than data-driven training tasks."
In the solving task, we categorize the independent variables into three distinct types based on their physical signif- icance. These are: spatial variables æ., which describe geometric features; the temporal variable Ił; and parametric variables Fp, which may include PDE coefficients, forcing terms, as well as initial and boundary conditions. Thus, a general PDE for physical problems can be represented a ppace-paaaeete--tine (S-P-T) problem.
In this paper, Eq. 12 is solved using a boosting algorithm. Moreover, instead of relying on stochastic optimization schemes, we subspace iteration to linearize Eq. 12, taking advantage of the sparsity in the resulting system of equations, and solve it using sparse direct solvers. Details about the data-free solver algorithm can be found in Ap- pendix D.
L(J u( (p. 0,
(11)
3.6. Inverse optimization
where C(-) denotes a general (nonlinear) differential op- erator; Ju(#,, pp .r2) is the approximated solution, com- monly referred to as the trial function in numerical analysis.
In many engineering applications, determining the optimal coefficients or design parameters from experimental data or underlying physics is essential. In such cases, INN-TD serves as an efficient parametric mapping tool from the input space to the output space, thanks to its sparse structure.
Methods for solving partial differential equations (PDEs) generally fall into two categories. The first involves directly minimizing the residual of the PDE in its differential form. The second projects the PDE loss onto subspace, evaluat- ing the loss via a weighted-sum residual in the integral form.
There are, in general, two different ways to tackle this prob- lem. In the first case, the spatial, temporal, and parametric variables are all treated as inputs to the machine learning model. Once the model is trained/solved,, one can easily leverage automatic differentiation to infer the parametric
Data-free solving code can he found in: https:/ //hub./ docker. com/r / oh wwwhnnhw/eeeeeccccco domo
5

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
inputs (Takamoto al., 2022). This approach requires the model to accurately learn the parametric mapping. In the second case, the physical parameters become trainable neu- ral network parameters. While they don't 't appear directly in the network, they influence the loss function through an additional residual (Zhang et al., 2022). The performance of this approach can sensitive to the contribution of each loss term and may require adaptive scaling of the loss func- tion components (Berardi al., 2025).
We use the Gaussian source function model the right-hand side (RHS) source term b(æ, P):
(- 2((x - x2)² (11 31)
b(ar, P) = D Pexp
2
(15)
where ro 0.05 is the standard deviation that characterizes the width of the source function and (Iz, Vi) is the i-th source center. We used nt 16 in this example as shown in Fig. 3.
In this paper, we adopt the first approach for INN-TD to solve inverse problems, as INN-TD can efficiently and ac- curately handle general parametric function mapping. The optimal parameters can be obtained by minimizing the fol- lowing loss function.
1.5
argmin C argmin ||] [ xp)) u' u'(xaa:F:)ll
#p
2pp
(13)
0.5
where "
|1- lle, is the L2 norm and is defined as | VE i=1 |xi u" (x., is the target spatial-temporal
00
field.
Figure Problem statement
4. Results
The simulation data are generated by running finite element analysis (FEA) with different parameters: k and P. We compare 4 different methods: MLP, SIREN, KAN, INN- TD. The results are summarized in Table. 1 where the root mean squared error (RMSE) is reported. Details on the model setup are provided in Appendix E. We can see that INN-TD outperforms in the training and test sets. Note that for KAN, the 30% and 100% training data cases use a batch size of 256 to prevent out-of -eroory issues, while the 10% training data case employs full-batch optimization, resulting in significantly lower error than the other two cases. Nevertheless, for all cases listed here, INN has higher accuracy compared to the other 3 different models.
In this section, we present the performance of INN-TD in three key areas: data-driven training, data-free solving, and inverse optimization. We compare its performance with other deep learning models including vanilla MLP, SIREN (Sitzmann et al., 2020), Kolmogorov-Armol net- works (KAN) (Liu et al., 2024), and tensor-decoomposition physics-informed neural networks (CP-PINN) (Vemuri et al., 2025). All of the cases were run on single NVIDIA RTX A6000 GPU.
4.1. Data-driven training task
4.1.1. TRAINING ON PARAMETRIC PDE SOLUTIONS
In this example, we aim to learn the solution a time- dependent parametric PDE that models heat transfer inside the domain. The input parameters spatial location (I. (I,y), time I, heat conductivity kt and source power P. The output is the temperature u. As a result, the function mapping of the parametric PDE can be written as (a, y, k, P.)) + u.
Table 1. Root Mean Square Error (RMSE) Comparison for Train- ing and Test Sets
Training (x 10
Test (x (x10 10:00)0)
Dataset
Model
Mean
Std
Mean
Std
MLP
4.180
0.873
3.660
0.498
SIREN
2,330
0.188
4,550
0.297
The governing parametric PDE can be written as:
Training 10%
KAN
4.600
4,840
0.323
INN. IND
0.213
0.009
0.748
0.911
ú(æ t) + Aau t) b(xx P) in Qu & Zr u(x t)ln = O, u1x 0) 0,
MLP
0.699
0.143
0.912
0.155
(14)
Training 30%
SIREN
KAN
4.710
1.240
0.161
1.120
4,490
2,020
0.996
0.197
INN-TD
0.154
0.003
0.154
0.006
MI.P
0.385
0.037
0.534
0.041
SIREN
1.170
0.121
1.660
0,200
defined in a spatial domain 22 = [0, 112, temporal domain Ω2 = (0, 0.04), and parametric domains Q2 = [1,4] 4] and
Training 100%
KAN
4.590
0.231
4.260
0.087
INN-TD
0.126
0.002
0.129
0.005
Qr = [100, 200).
6

---

Interpolating Neural Network-Tensor Decompasition (INN-TD)
Table. lists the detailed performance of each different method in Case 1. INN-TD stands out due to its better performance in terms of accuracy and computational effi- ciency. For instance, in the 2D setting, INN-TD achieves a low relative 1.2 norm error of 1.754 x 10-8 in just 0.81 seconds, significantly outperforming other methods in both accuracy and speed. As the dimensionality increases to and 10, INN-TD maintains its high accuracy with errors of 1.659 x 10-$ and 1.238 x 10-$, respectively, while still being computationally efficient. CP-PINN and KAN are able to tackle high-dimensional problems. However, since the total number of collocation points will increase expo- nentially with the increase of dimension, they still suffer from expensive computational costs for high-dimensional problems. This demonstrates the robustness and scalability of the INN-TD method, making it promising approach for solving high ligeedimensional PDEs.
38 1.
10
(a) Domain size a [0, 1/2
(b) Domain size a E [0, 12?
Figure 4. Analytical solution for 2D cases.
4.2. Data-free Solving task
4.2.1. SOLVING HIGH-DIMENSIONAL PDE
In this example, we use INN-TD as a data-free solver to directly solve high-ding sigh-dimensional Poisson's 's equation with different source terms and compare its performance in terms of accuracy and speed against CP-PINN and KAN, Detailed model setups can be found in Appendix E.
Table 2. Comparison of different data-free solvers with domain size [0, 1] INN-TD outperforms other models in both accuracy and speed. The mean values for error and wall time are reported. Model
dim
# collo-
#
Relative L2 norm
GPU
The multi-dimensional Poisson's ' equation is written as:
cation points each
model param- eters
error
wall
time
(sec)
dim
Au(z1, 22, xp) = f(1 .2.,,7.7 22.,
(16)
256
1.754 x 10
0.81
INN-TD
1,600
1.659 x 10-8
6.88
10
6,400
1.238 x 10
57.43
where (-)) is the Laplace operator which is defined as: (() = ()() + O²() d(), D denotes the total num- कzÿ Jzz +, dzz"
32
96
4.63 10
60.68
CP-PINN
2.64
169.2
32
1,200
5.90
130
ber of dimensions.
32
348
9.44
56.16
KAN
128
348
1.81
10
58.88
Point-wise relative 1.2 norm error is used to measure the accuracy since INN-TD is also compared with collocatio- based machine learning method:
12
624
1.68 x 10
842.1
Table. 3 provides a performance comparison of different
models in Case 2. It is evident that the accuracy of both CP- PINN and KAN significantly decrease compared to Case 1, due to the increase in domain size. Despite utilizing more collocation points, the relative L.2 norm error for these two methods does not converge to a smaller value, indicat- ing possible limitations in applying them to industrial-level problems requiring high accuracy and solver convergence. In contrast, INN-TD demonstrates significantly better accu- racy, outperforming other methods by orders of magnitude while requiring much less computational time.
1/2
K
c= Σ ((((())) u exact (xx)]
Σ
ezant (xx)²
(17)
where K is the total number of test data; u mract (ack) is the analytical solution. We let (xx) n² Ei sin (5x))
and the corresponding analytical solution is:
u exact
(x)
sin
(18)
2
Furthermore, we evaluate the convergence of the INN-TD data-free solver for Case 1. As illustrated in Fig.5 (a), INN-TD demonstrates favorable convergence properties, with reduced error achieved as the model complexity in- creases. This convergence property enhances confidence in the model's 's accuracy, enabling the design of the network to align with the anticipated precision.
Two cases are examined here. In Case 1, the domain size is [0, _j_ while Case 2 explores larger domain size [0, I2)P. The analytical solution is shown in Fig. 4 for both cases when D 2. As evident from the figure, Case 2 is more challenging to solve due to the presence of higher-frequency signals in its analytical solution. It's known that vanilla MLP has spectral bias and thus poses challenges to learn Case 2 accurately and efficiently (Rahaman al., 2019).
In addition to this example, three more examples are in- cluded in Appendix E.2 to further demonstrate the efficiency and accuracy of INN-TD as data-free solver for tackling
7

---

Interpolating Neural Network-Tenso Decomposition (INN-TD)
Table Comparison of different data-free solvers with domain
size 10, _ INN-TD outperforms other models in both accuracy and speed. The mean values enror wall time are reported. Model
Relative
Relative L2 norm error
10
cation
model
error
wall time
2.00
points param-
in each
dim
10
2
256
4.77 x 10
0.22
1.98
INN-TD
5
1,600
3.97 x
7.04
6,400
3.71 10
59.75
1.87
10-7
216
$24, p=1
CP-PINN
2
2
2,048
1,024
216
296
3.2 x 10
x 10-2
261.6
292
pw2
3.49
2
32
348
7.8 x 10-1
60.68
2x
3x
10
10*
KAN
2
128
348
8.0 x 10
72.6
Number of model parameters
2
128
3,188
2.8 x 10
119.7
(a)
5=1,
5u4, pul
large-scale high-dimensional PDEs. In particular, we show INN-TD doesn't exhibit the failure modes of PINN when solving the Helmholtz equation (Wang et al., 2021).
5=2, pm2
Computation (5)
10a
9x 10
4.2.2. SOLVING GENERAL PAREE-ARR -TME (S-P-T) PDE
8 10
A distinct advantage of machine Saarnnng-based data-free solvers is their ability to naturally handle parametric PDEs by treating parametric variables as inputs. In contrast, stan- dard numerical solvers like FEM must rerun simulations for each different set of parametric inputs. In this example, we demonstrate that INN-TD effectively handles parametric PDEs by utilizing a sance-parameter-tmee (S-P-T) interpo- lation. We consider solving the time-dependent parametric PDE described in Eq. 14, treating it 3D spatial problem x (Ir, U, =) with 2D parametric inputs p (k, P) and ID temporal input 2, = t.
10
xx10' 10'
3x 4x model parameters
6 10
(b)
8.8 x 10
$ud, p=1
5w2, pm2
8.6 10
8.4 10
VRAM
8.2 x0'
The new source function can be written as:
8x100
(- v,) ) + (u 3:)3)
2
b(x, Ei P
1:2d0 (19)
r
7,8x10 10
7.6 10?
where dg is the depth of the source and do = 0.5 in the cur- rent example; 1 122d, is the indicator function where 1,2d, equals 1 if 2 do 0 iif= < dg- Therefore, this PDE can be interpreted as modeling heat transfer with multiple fixed heat sources, each characterized by a radius ro and penetration depth dg. as shown in Fig. 6.
10"
x 10 10 model parameters
6 10'
(c)
Figure 5. Solving the 2D Poisson's Eq. using INN-TD with dif- ferent number of model parameters and hyperparameters and p. a=20 used for all cases. The statisties are obtained by repeating each case for 10 times and the shaded area represents standard deviation (a) convergence of INN-TD using different and p (b) statistics of total computational time (c) statistics of GPU VRAM usage
This problem is solved using INN-TD using the algorithm listed Algorithm by discretizing each input dimension with 101 grid points and utilizing 100 modes. The total solving time takes 155
single NVIDIA RTX A6000 GPU. Fig. 7 compares the INN-TD solution with FEM when h: = 1.03, P = 101, 0.95.
with different realizations of parametric inputs (k, P). The relative L2 norm error for INN-TD 3.38 x 10-3 De- tails about the error computation can be found in Appendix E.2.4.
To quantitatively assess the accuracy of the solution ob- tained from INN-TD, we measured the accuracy of INN-TD against the high-fidelity simulation by running implicit FEM
8

---

Interpolating Neural Network-Tenso Decomposition (INN-TD)
Multiple heat sources
e1. |[ Fri Fp; u' (x rz)ll2
(21)
(Fa: | 24)
The error metrics of this example are summarized in Table. 4. As shown in the table, INN-TD effectively recovers the optimal parametric inputs with high accuracy. Details on the model setups be found in Appendix E.2.5.
Table Error for inverse optimization task.
Error Type
Mean
Standard Deviation
Figure 6. S-P-T data-free solver problem
CLg
2.18 x 10-3
5.64 x 10
6k
2.76 x 10-3
4.64 x 10-4
2.1e+02
Ep
2.62 10
3.11 x 10-4
180
160
120
5. Conclusion
130
3
80
In this paper, we introduced INN-TD as an efficient and accurate function approximation for large-scale, high- dimensional problems governed by PDEs. By combining the strengths of machine learning techniques and classical numerical algorithms, INN-TD demonstrated high accuracy in data-driven training. data-free solving, and inverse opti- mization tasks for problems where the underlying physics are described by PDEs. Particularly in the solving task, INN- TD achieves orders of magnitude better accuracy compared to other machine learning models. Convergence properties have been observed where more model parameters result in higher accuracy. Furthermore, INN-TD efficiently scales to nigher-dimensione. problems with extremely fast solv- ing speeds. Lastly, due its interpretable nature, we can tailor the structure of INN-TD according to the required resolution in physical problems. In conclusion, by unifying training, solving, and inverse optimization within a single framework, INN-TD offers a seamless approach to address- ing challenging, large-scale physical problems that demand high precision and rapid solution speeds.
60
INN-TD
40
20
FEM
Figure 7. Comparison of INN-TD and FEM solution for k 1.03, P 101, t 0.95
More data-free solving problems can be found in Appendix F.2.
4.3. Inverse optimization task
In the previous section, we have shown INN-TD can effec- tively capture complex S-P-T function mapping for both data-driven training and data-free solving tasks. Conse- quently, for the inverse problem, we first train (solve) the function mapping from S-P-T continuum to output using the data-driven trainer (data-free solver). Then we recover the unknown parametric coefficients by minimizing the differ- ence between prediction and measurements using automatic differentiation.
Acknowledgment
The authors would like to greatly thank Prof. Thomas J.R. Hughes from the University of Texas, Austin for his insight- ful feedback and suggestions on the current work.
As an example, we use the same PDE as in section 4.2.2 to il- lustrate the performance of INN-TD for the inverse problem. In this task, we aim to find the optimal parametric inputs (ks, P) obtain the target spatial-temporal field u² (xx, xz).
Impact Statement
We adopt the following 2 metrics analyze the accuracy. The error of identified parametric input is defined as:
INN-TD lies at the intersection of artificial intelligence (AI) and traditional simulation approaches. By embedding in- ductive biases inspired by numerical analysis directly into the network architecture, INN-TD effectively tackles the challenges of low accuracy and high computational cost as- sociated with current deep learning models when applied to high- dimensioonnl,, large-scale physical problems governed by PDEs.
(20)
The error of the prediction is defined as the relative L2 norm error between estimated output and target output.

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
References
Guo, J., Park, C., Xie, X., Sang, Z., Wagner, G. J., and Liu,
W. K. Convolutional hierarchical deep learning neural networks-tenso decomposition (c-hidenn-td): scalable surrogate modeling approach for scaeescale physical sys- tems. arXiv preprint MrXiv:249.003222 2024.
Ahmed, T., Raja, H., and Bajwa, W. U. Tensor regression using low-rank and sparse tucker decompositions. SIAM Journal on Mathematics of Data Science, 2(4):944-966 2020.
Guo, J., Domel, G., Park, C., Zhang, H., Gumus, O. C., Lu, Y., Wagner, G. J., Qian, D., Cao, J., Hughes, J., et al. punssoddccompositiion.o (taps) modeling for ultra large-scale simulations. arXiv preprint arXiv:2503. 13933, 2025.
Bachmayr, M. Low-rank tensor methods for partial differ- ential equations. Acta Numerica, 32:1-121, 2023.
priori surrogate
Berardi, M., Difonzo, F. V., and Icardi, M. Inverse physics- informed neural networks for transport models in porous materials. Computer Methods in Applied Mechanics and Engineering, 135:117628, 2025.
Huang, O., Saha, S., Guo, J., and Liu, W. K. An introduction to kernel and operator learning methods for homogeniza- tion by self-consistent clustering analysis. Computational Mechanics, (2(1):195-21) 2023.
Beylkin, G. and Mohlenkamp, M. J. Algorithms for nu- merical analysis in high dimensions. SIAM Journal on Scientific Computing, 26(6):2133-2159, 2005.
Hughes, J. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2003.
Bungartz, H.-J. and Griebel, M. Sparse grids. Acta numer- ica, 13:147-269, 2004.
Kharazmi, E., Zhang. Z., and Karniadakis, G. E. hp-vpinns: Variational physics-ifformed neural networks with do- main decomposition. Computer Methods in Applied Me- chanics and Engineering, 374:1 13547, 2021.
Cohen, A., Vere, R., and Schwab, C. Convergence rates of best n-term galerkin approximations for a class of el- liptic spdes. Foundations of Computational Mathematics, 10(6):61--666, 2010.
Kolda, G. and Bader, B. W. Tensor decompositions and applications. SIAM review, 51 2009.
Colbrook, M. J., Antun, V., and Hansen, A. C. The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smale's 18th problem. Proceedings of the National Academy of Sciences, 1 119 (12):22107151119, 2022.
Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., and Mahoney, M. W. Characterizing possible failure modes in physics- informed neural networks. Advances in neural information processing systems, 34:26548-26560, 2021.
Cybenko, G. Approximation by superpositions of a sig- moidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
Li, H., Knapik, S., Li, Y., Park, C., Guo, J., Mojumder, S., Lu, Y., Chen, W., Apley, D. W., and Liu, W. K. Convo- lution hierarchical deep-learning neural network tensor decomposition (c-hidenn-td) for high-resolution topology optimization. Computational Mechanics, 72(2):363-382, 2023.
Dolgov, S. V., Khoromskij, B. N., and Oseledets, V. Fast solution of parabolic problems in the tensor train/quantized tensor train format with initial applica- tion to the fokker- fokker-planck equation. SIAM Journal on Scientific Computing, 6306)):3016-33000 2012.
Li, Z., Kovachki, N., Azizzadeneshheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equa- tions. arXiv preprint
Doshi- Velez, F. and Kim, B. Towards a rigorous sci- ence of interpretable machine learning. arXiv preprint arXiv:1702.088 08608, 2017.
2020.
Liu, W. K., Li, S., and Park, H. Eighty years of the finite element method: Birth, evolution, and future. Archives Computational Methods in Engineering, 29(6):4311- 4453, 2022.
Gao, H., Sun, L., and Wang, J.-X. Phygeonet: Physics- informed geometry-adaptive convolutional neural net- works for solving parameterized steady-state pdes irregular domain. Journal of Computational Physics, 428: 110079, 2021.
Liu, 7.,, Wang, Y., Vaidya, S., Ruehle, F. Halver- son, J., Soljačić, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756 2024.
Grossmann, T. G., Komorowska, U. J., Latz, J., and Schönlieb, C.-B. Can physics-informed neural networks beat the finite element method? IMA Journal of Applied Mathematics, pp. hxae011, 1, 2024.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via deeponet based on the
10

---

Interpolating Neural Network-Tensor Decompasition (INN-TD)
universal approximation theorem of operators. Nature machine intelligence, 3(3):218-22) 2021.
Sidiropoulos, N. D., De Lathauwer, L., Fu, X., Huang, K., Papalexakis, E. E., and Faloutsos, C. Tensor decomposi- tion for signal processing and machine learning. IEEE Transactions on signal processing, 65(13):355133882 2017.
Lu, Y., Li, H., Zhang, L., Park, C., Mojumder, S., Knapik, S., Sang, Z.., Tang, S., Apley, D. W., Wagner, G. J., al. Convolution hierarchical deep-learning neural networks (c-hidenn); finite elements, isogeometric analysis, tensor decomposition, and beyond. Computational Mechanics, 72(2):333-362, 2023.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with peri- odic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
McGreivy, N. and Hakim, A. Weak baselines and reporting biases lead overoptimism machine learning for fluid- related partial differential equations. Nature Machine Intelligence, pp. 1-14, 2024.
Takamoto, M., Praditia, T., Leiteritz, R., MacKinlay, D., Alesiani. F., Pflüger, D., and Niepert, M. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35: 1596-1611, 2022.
Melenk, J. M. and Babuška, The partition of unity finite element method: basic theory and applications. Computer methods in applied mechanics and engineering, 139(1-4); 289-3144 1996.
Vemuri, S. K., Büchner, T., Niebling, J., and Denzler, J. Functional tensor decompositions for physics-informed neural networks. In International Conference on Pattern Recognition, pp. 32-46. Springer, 2025.
Park, C., Lu, Y., Saha, S., Xue, T., Guo, J., Mojumder, S., Apley, D. W., Wagner, G. J., and Liu, W. K. Convolu- tion hierarchical deep-learning neural network (c-hidenn) with graphics processing unit (gpu) acceleration. Compu- lational Mechanics, (2(2):383-409,2003
Wang. S., Teng, Y., and Perdikaris, P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 4365):3305--33881 2021.
Park, C., Saha, S., Guo, J., Xie, X., Mojumder, S., Bessa, M. A., Qian, D., Chen, W., Wagner, G. J., Cao, J., al. Engineering software 2.0 by interpolating neural net- works: unifying training, solving, and calibration. arXiv preprint arXiv:2404. 10296, 2024.
Wang. S., Yu, X., and Perdikaris, P. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 10768, 2022.
Zhang, E., Dao, M., Karniadakis, G. E., and Suresh, Anal- yses of internal structures and defects in materials using physics-informed neural networks. Science advances, 8 (7)caabk0644, 2022.
Pfaff, T., Fortunato, M., Sanchez- Sanchez-Gonzalez, A., and Battaglia, W. Learning mesh-based simulation with graph networks. arXiv preprint 20Xiv20010003409, 2020.
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. On the spec- tral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMLR, 2019.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics- informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Rathinam, M. and Petzold, L. R. A new look at proper orthogonal decomposition. SIAM Journal on Numerical Analysis, 41(5):1893-1977) 1925, 2003.
Rathore, P., Lei, W., Frangella, Z., Lu, L., and Udell, M. Challenges in training pinns: loss landscape perspective. In Proceedings of the 41st International Conference on Machine Learning, 1CM..24. JMLR.org, 2024.

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
A. C-HiDeNN interpolation theory
cnvolutional-Hierarchhh Deep Neural Network (C-HiDeNN) interpolation theory synergizes the advantages of finite
element interpolation, mesh-free interpolation, and machine learning optimization. (Lu al., 2023; Park al., 2023). ID C-HiDeNN interpolation can be written in the algebraic form
Ju(Ø)= EN((v) iEA
Ni(1) Σ W00 (,
Σ keAs
Na(2; a,
(22)
JEA;
where (xr is the standard finite element basis function node i; w (x; 8 a, a,p) is the convolution patch function at node j defined on the support domain A. The double summation can be combined single summation over elemental
nodal patches defined A Use A².. We can also interpret Eq. 22 partially connected MLP, shown in Fig. 8.
(a) 1D C-HiDENN
1-3,
5-2
= 2,p
1
X1-3
X1-2
X/-1
Xy
X
X4+2
X/+3
X1+4
mapping
A nodes within linear finite element
patch size, integer
Al paten domain at node with patch size
dilation parameter
AC Userr 45 patch nodes of element with patch size
reprocucinn polynomial Integer
(b)
(c)
Padd (crw))
Ry
G.
(x
Number convolution neurons A
Convolution
patch function
R
Ry-R.tx
Convolution
(5)
patch functions
mapping
Lym'
WC2 (3)
Aym
Ryar
Ryzs
A yusrt
Polynomial order p determines
Wg((5)
E
Ay-xo
Grintm)
(x%)
neurons (-
A y
A Aymx
Ty-
wi W convolution Interpo. node /
actyvtion
redxil redial basis function centered node
faxtion, y f64
+3
is
6: moment matrix
*h; nodal coordinates of nodes 4g
weights and biases
Figure (a) Convolution patch in ID C-HiDeNN shape function (b) Construction convolution patch function (c) C-HiDeNN shape function as MLP with 3 hidden layers. This plot is adopted from (Park et al., 2024).
In Fig. 8, the convolution patch function W:() (i;; 8, a. p) controlled by three hyperparameters: patch size s that controls nodal connectivity, dilation parameter a that normalizes distances between patch nodes, and reproducing order p that defines types/orders of activation functions to be reproduced by the patch functions. C-HiDeNN can adapt to these hyperparameters node by node using optimization, rendering an adaptable functional space without altering the number of nodes or layers.
Unlike the traditional black-box MLP, C-HiDeNN can be interpreted as tailored MLP structure that preserves several desired properties (inductive biases) that are particularly important in numerical analysis: locality, Kronecker delta property (Hughes, 2003), partition of unity (Melenk Babuška, 1996), and easy for integration (Hughes, 2003). These attributes enable C-HiDeNN efficiently model complex physical problems with high accuracy.
12

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
Table 5. Comparison of ML. and C-HiDDeNN
Properties
MLP
C-HiDeNN
Boundary/initial condition
Penalty term in the loss func- tion (Raissi et al., 2019)
Automatic satisfaction due to
Kronecker delta (Lu et al., 2023)
Convergence and stability
Stochastic and not guaranteed
Shown for different PDEs (Guo et al., 2024)
(Colbrook et al., 2022)
Numerical integration
Quasi-Monte Carlo integra- tion (Kharazmi al., 2021)
Gaussian quadrature (Hughes, 2003)
Interpretability
Black-bo model
Fully interpretable
B. Interpretability of INN-TD
Unlike most black-box machine learning models. INN-TD has a fully interpretable architecture. As shown in Fig. 9, INN-TD can be interpreted as a specially pruned MLP where the first hidden layers represent locally supported linear finite element basis functions. The 3rd hidden layer reconstructs the C-HiDeNN interpolation function with higher-order smoothness which is controlled by hyperparameter patch size $ and reproducing polynomial order p. The 4th hidden layer leverages tensor decomposition counter the curse of dimensionality. As result, INN-TD's learning parameters interpretable nodal values of TD components, in contrast to the opaque weights and biases of neural networks.
To demonstrate the benefits of INN's interpretability. we solve the following 2D Poisson's 's equation in Ω = [0, 100]² with local source function.
02
82
+
= f(2,z)
(23)
ix
0u3
The local Gaussian source function f(x, y) is defined as:
f(x,y) = 0.064(z - -0.04( 44)² e 0.044 76) + 0.064( - 20 -0.0(((-0000) 20)² 00000000000000000000000000
0004
+0.064(y e 0.04(z- 0.04(>-48)) -0.04(y 55) 0.064 25)² 0.04(a -0.04(y-55)
(24)
1166 0.04(2-20)² -004((y25))
1.6c 1.6c 0.04(a--000000 40)2 -504((--55)
The exact solution to this problem is:
(x12
25)?
(a)
70)
(I,)) 10c 23
26
+
10c
26
25
(25)
The Dirichlet boundary condition is ulon u (),y))
To efficiently solve this problem, we leverage the interpretable locally supported basis function in INN-TD. As can be seen from Fig. 10 (a), a nonuniform mesh is used to accommodate the local source function: a fine mesh is used for the source function region, whereas a coarse mesh is used for other regions. Moreover, since the nonlinearity of the solution expected to be localized near the source function region, larger and p can be used for these local regions to improve the accuracy. The absolute pointwise error with respect to the exact solution plotted in Fig. 10 (d) where the order is around 10" 10-6, -6
C. INN-TD trainer
Two different training schemes can be used for INN-TD trainer. The first method is based on the boosting algorithm. The core idea behind boosting is to convert a set weak learners into strong learner. A weak learner a model that performs slightly better than random guessing. The boosting process involves sequentially training multiple weak learners, each trying to correct the errors made by its predecessor.
We define the weak learner for mode m as (mmm)((2)) U()),, where ((m) = [u(mm) ; ,u2 ..... u (mn) is the model parameter for mode m. The strong learner Jy(2; U) can be obtained by adding weak learners for each mode, and we define U (U(1), U22), ( UM)]
(m)
Jy U) M F(m) (2r; (mm)
(26)
13

---

Interpolating Neural Network-Tenson Decomposition (INN-TD)
al. Computational Mechanica
72, (2023): 333-362.
1
C-HiDeNN
0
8,(%))
E N. (K.) MA
(x,)
Modes
Zhang, L. et al. Computational
Mode M
Mechanics 67 (2021); 207-230. Alg ReLu
elemmnnt
Time 1eg
elehnnnt
A
A
A
Pllmemnee
Input variables
HiDeNN
32
Tiee
ITe ddms
element
0
element
xp
element 1
u
Σ
Nua (x)
elc.t
Mode 1
m= d=1
Figure 9, Interpretable structure INN-TD
The loss of model prediction and training data "y" is defined using the mean squared error:
C = K (Jy(Ei - -
(27)
Assuming the previous M -11 weak learners have been obtained, and we aim to learn the new weak learner (M) y (MM).. Eq. 27 can be written as:
1 K
Σ
(DM=T y( Ulml) yais ((DD) y²)
(28)
Consequently, the goal is to learn to model parameters (M) for weak learner (M) y(x; UMM) using optimization schemes.
argmin L 1
(DM-) y U(m) + g(M) U(M) -2
(29)
UTM
K.
The complete boosting algorithm for INN-TD trainer is shown in algorithm 1.
Instead of gradually enhancing the accuracy of the model by adding weak learners, one can also predefine the total number of modes M required in the model and optimize the unknown parameters in the model in an all-at-once fashion as shown in Eq. 26. Consequently, we treat the total number of modes as one additional hyperparameter. The all-at-once training
14

---

Interpolating Neural Network-Tense Decomposition (INN-TD)
100
INN-TD design
130
INN-TD solution
Ecactioaluionn
Absolute error
8.8
line mesh
80
7.7
80
1.7
90
4.
66
60
5.5
Gaussian
444
(20, 25)
20
18
11
0.
100
0
20
00
100
20
100
0.0
0
40
80
0.0
(a)
(b)
(c)
(d)
Figure 10. Interpretable design of INN-TD: (a) ID meshes used in INN-TD are only refined at the location where nonlinearity happens with larger and used to achieve higher-order smoothness (b) IN-TDD solution (c) Exact solution (d) point- wise absolute error
Algorithm 1 INN-TD trainer: boosting
Define maximum number of weak learners (modes) M, maximum epoch number epochmear grid for each dimension æd Initialize model J (x) 0 for (epoch loop) epoch = to epochmaz do for (boosting loop) m = 1 to Mmax do Initialize solution vector (mm) for the m-th weak learner. Optimize Eq. 29 using optimization schemes such as Adam
Update learner: J (xx Ty (x))) ((m) (a:; U(m) Check loss
end for end for
strategy has been shown to generally lead to better accuracy compared to the boosting strategy when the total number of modes is the same.
The complete all-at-once algorithm for INN-TD trainer is shown in algorithm We use this algorithm for all of the training experiments presented in the paper.
Algorithm 2 INN-TD trainer: all-at-once
Define total number of modes M, maximum epoch number epoch grid for each dimension a Initialize model parameters U = (1) ( ..... [.M)
for (epoch loop) epoch = to epochmaz do Optimize Eq. 26 using optimization schemes such as Adam. Check loss
end for
D. INN-TD solver
Boosting algorithm is adopted for INN-TD solver. Such method is known as proper generalized decomposition (PGD) the field of computational mechanics (Li al., 2023). A similar all-at-once approach named priori tensor decomposition is also available (Guo al., 2024). Different from the training approach where the loss is computed in point-wise manner, we compute the PDE loss in weighted-sum residual format. Without loss of generality, we focus on the discussion of the scalar solution field. Vector and higher-order tensorial solution fields can be treated in a similar way. Assume PDE can be written as:
Lu(
(30)
15

---

Interpolating Neural Network-Tensor Decompasition (INN-TD)
where L is general (nonlinear) differential operator, u(x) is the D-dimensional PDE solution depending independent variables (211 20,n TD). The INN-TD approximation to the solution can be written
M
Ju(x)= Σ m=1
(1
(mn) (2(22)u (m)
N (DDD) (m)
(31)
In the context of solutions time-dependent parametric partial differential equations (PDEs), the independent variables (21, .2,..................... ................................ D) can be divided into three distinct categories: spatial variables 2. parametric variables 2p. and temporal variables d₁. Spatial variables a, define the spatial coordinates relevant the problem. Parametric variables Pp serve as additional coordinates and can represent PDE coefficients, initial conditions, boundary conditions, and geometry descriptors. Temporal variable 22 represents time.
The weighted-sum residual form is adopted to solve PDEs.
/ [(Ju(E))((T
(32)
where ((])( is the so-called weighting function. Depending the form of the PDE, different choices of ([]u(a)) can be adopted. For instance, if the Dirac Delta function 8(x æz) (6 here refers to the Dirac Delta function) is used for the weighting function where æ; is the collocation point, Eq. 32 becomes the differential (strong) form: Lu(x) 0. In this paper, we adopt the Galerkin formulation, where the function space of the weighting function is the same as our approximation. Moreover, integration by parts can be used to alleviate the differentiability requirement on the approximated solution (Hughes, 2003).
In the boosting solution scheme, the solution is obtained incrementally in mode by mode fashion:
M-1 Ju(x)= Σ (T)
(m)
N2 (D(D)) (11))u N2(22)
ND(ED)UZD (33)
m=1
where we neglect the superscript M for the last mode M that we are solving. As a result, the corresponding weighting function can be written as:
6Ju(x)= = N1(21) N2(22) urz
ND(ED)uzn+ ND(ED)uent
(11)) Ne(r2)ōurz
(34)
(z12)u Ne(
(D(ED)UED
(1 N2(12)
ND(
Based on Eq. 34, we further subspace iteration to linearize the problem. In subspace iteration, sequentially alternate 1 unknown variable while treating others known constant values. In this case, the variation for other unknowns will be O. Taking subspace iteration in иj as an example, Eq. 34 becomes:
Tn)T)(()) = (o:)ou, N2(x22)uz ..... ND(ED)
(35)
Plugging Eq. 33, 35 into Eq. 32 and integrating along each ID dimension, the matrix equations for xı dimension can be derived:
Ag, Uz Qa
(36)
where ur is the solution vector for the current dimension at the current mode; Az, is banded sparse matrix as in the standard finite element method thanks to locally supported basis functions; Q., is 1D vector.
Similarly, we can sequentially alternate other dimensions and obtain the corresponding linear system of equations:
Az, Uks Q Qua
(37)
16

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
where d = =1,... D; ue is the solution vector for the current dimension the current mode; Asa is banded sparse matrix; Qz is ID vector. As result, Eq. 37 can be efficiently solved using sparse matrix solvers. We keep iterating Eq. 37 until the variation of solution vector for each dimension is within tolerance. In practice, 3-5 iterations will yield good accuracy.
After the current mode is solved, we can introduce additional modes to further enhance the accuracy of the INN-TD model. This iterative process will be continued until the results meet the predefined accuracy criteria.
The complete boosting algorithm for INN-TD solver is shown in algorithm 3.
Algorithm 3 INN-TD solver: boosting
Define total number of modes M; maximum number of iteration itermaz; tolerance; grid for each dimension Zd Initialize solution I [=(=== =
Initialize solution vector u. for (boosting loop) 1 to M do for (subspace iteration loop) iter = 1 to iter do for (dimension loop) d = 1 to D do
Update Az, and Q.
Solve Ar,, uud (m)
Qn using sparse matrix solver
end for
Check convergence
end for
Ju((x) = Tu Ju(x)= + (11)) (m)
(2(22)
(m)
ND(D))usp (mr)
Check Error
if Error Tolerance then Break
end if
end for
E. Detailed model setups
E.1. Data-driven training examples
E.1.1. MODEL SETUPS OF TRAINING ON PARAMETRIC PDE SOLUTIONS
This section presents the hyperparameters for training models on the PDE dataset. For the MLP model, we set the initial learning rate to 0.001, the batch size to 128, and use the Adam optimizer. Training runs for a maximum of 500 epochs with early stopping based on the validation error. If the validation error increases for 10 consecutive epochs, the model training will be stopped. We also apply an adaptive learning rate scheduler, reducing the learning rate by 10% after each epoch. The MLP model has 5 hidden layers, each having 50 neurons.
For the SIREN model, we use almost the same hyperparameters the MLP model. But reduce the initial learning rate to 0.0001 to avoid loss explosion. We also increase the model complexity of the SIREN model. The model consists of 4 hidden layers, each containing 100 neurons.
For the KAN model, we set the layer widths to |5, 5, 1] for the 10% training data case, given that the input and output dimensions are 5 (x. y, t, ks p) and (u), respectively. We use 10 grid intervals.third-ordes third-order of piecewise polynomials, and the LBFGS optimizer with 2, 000 steps and full-batch training. For the 30% and 100% training data cases, full-batch training infeasible due to the GPU memory limitation, so we the batch size to 256. With smaller batch size, we increase the layer widths to [5, 10, 10, 1] and extend the number of training steps to 3,000. Notably, training loss can easily explode to NaN and increasing batch size does not change the final error too much.
The INN-TD model uses 60 modes with 40 elements while the hyperparameters are: patch size 4, dilation parameter a = 20, and polynomial order = 1. Thus, there are a total of x 60 x 41 = 12, 300 trainable parameters. We use the batch size of 128 for 100 epochs with the same early stopping criteria adopted in MLP. The Adam optimizer is adopted with a learning rate of 0.0001.
17

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
E.2. Data-free solver examples
E.2.1. HIGHI-DIMENSIONL PDE
This subsection has detailed information discussed in section 4.2.1. Four different models, namely, INN-TD, PINN, CP-PINN, and KAN, were used as data-free solvers. The following high-dimensional PDE is solved.
Au(zI, 22 .. ID) f(11, 22, ... DD)
(38)
In case I, we adopt the following analytical solution:
u urratt (xc)
sin
2 T/s
(39)
The corresponding right-hand side (RHS) is:
2
f(x)
Iz
(40)
Detailed performance comparison of different models is listed in the tables below.
Table 6. Case I: domain size [0, 1 (each case is repeated 10 times obtain the statisties)
Model
Dim. (D)
grid pts in each dim
modes
# model params.
Mean norm error w/ std.
GPU
Mean YRAM usage
points
lime w/ std.
(MB)
2
32
256
1.754
10
+ =1x 10
0.81 0.10
760 1 0
INN-TD
(4 iterations)
10
5
32
32
1,600
6,400
1.659
1.238
10
1,6 12
1.2 10
57.43 + 0.41
6.88
0.03
760 + 0
760 1: 0
2
32
96
4.63 x 10 7.14
1071
60.68 + 0.53
880 + 0
CP-PINN
(80 K epochs)
2
128
96
2.61
x 10 10-3 +
169.2 + 4.83
882 + 0
32
1,200
5.90 10-3
6.91
2, 130 18.6
3018
2
32
5
348
9.44 10
8.37
10
56.16 + 0.58
866 + 0
(50 steps)
KAN
2
128
5
348
4.84 x 10
8.95
x 10
58.88 1 0.58
1, 160 1: 0
5
12
624
1.68
10
2.45
10
842.1 +: 3.94
0,988 +
Table 7. Case domain size x 00, 12 (each case repeated times obtain the statisties)
Model
Dim.
colloc. points
grid pts in each dim
modes
model params.
Mean rel. norm std.
Mean GPU wall
Mean GPU
time std. (6)
VRAM usage
w/ std. (MB)
32
4
256
4.77
10 +
10-8
0.22 0.006
760
INN-TD (4 itcrations)
32
10
1,600
3.97
10
4.02
10
7.04 + 0.04
760 +
10
32
10
6,400
3.71 x
10
2.69
10
59.75 0.39
760 +
CP-PINN
322
4
216
0.025 0.004
87.6 + 6.53
882
(80 K epochs)
1,024
4
216
0.021 + 0.012
261.6 + 7.79
1,014 +
2.018
4
296
0.032 + 0.007
292 + 10.3
1,014 +
32
348
0.78 0.40
60.68 0.53
866
(50 steps)
1,024
348
0.50 + 0.27
72.6 = 1.19
19,948 +
128
3,
0.28 1: 0.08
119.7 2.05
2,486 +:
We also investigated the performance of four different models for the following analytical solution as in Case 2.
D
(+)
II
(sin(TTa))
(41)
d=1
For 2D cases, the analytical solutions are shown Fig. 11.
18

---

Interpolating Neural Network-Tenson Decomposition (INN-TD)
1.0
1.00
0.75
0.6
0.50
u
0.4
0.25
0.00
0.2
-0.5
0.50
0.0
1.00
0.8
12
0.0
0.6
10
0.2
0.4
0.4
0.6
0.2
4
1.0
X2
X1
X1
10
12
X2
(a) Domain size a E [0, 1]²
(b) Domain size E [0, 12
Figure 11, Analytical solution for 2D cases.
The corresponding RHS can be written as:
f(x) - II (sin (TTa))
(42)
Similarly, we studied 2 subcases with different domain sizes. The detailed comparison is shown in the table below.
Table Case 2: domain size [0, 1 (each case repeated times obtain the statistics)
Model
Dim. (D)
colloc. points
ggrid pts each dim
modes
# model params.
Mean rel. norm error w/ std.
Mean GPU wall time w/ sld. (s)
Mean
VRAM w/ std. (MB)
INN-TD
iterations)
5
2
32
32
160
64
3,03 x 10
1.23 x 10
1 8.65 x 10-14
+ 1.31 10-14
0.81 0.10
1.01 0.02
760 1 0
760 + 0
10
32
320
2.42 x 10
18.1
10
3.62 0.06
764 1: 0
CP-PINN
2
32
96
4.13 10
1.00
10
171.68 + 15.3
880 1 0
(80 K opochs)
2
128
96
5.17 10 +
1.38
10
109.2 = 14.83
882 + 0
5
32
10
1,200
8.10 x
10-00-1100 1.45
x 10
2,102 + 73.8
3,018 + 0
2
322
348
8.03
+ 6.90
10
57.5 0.48
866 + 0
KAN (50 steps)
2
1282
5
348
9.13 10
+ 1.90
10
61.0 + 0.35
1,100 = 0
5
12
5
624
0.59 + 0.0054
872.8 + 3.71
9,988 0
Here we investigate the convergence of INN-TD for the 2D case with domain size Ω E [0, 12 The details are shown in Fig. 12. INN-TD shows convergence property where larger number of model parameters leads to better accuracy. Moreover, the convergence rate can be controlled by adopting different combinations of hyperparameters and
E.2.2. SOLVING THE HELMIOOLTZ EQUATION
In this example, INN-TD is used to solve the Helmholtz. equation as shown below. In (Wang et al., 2021), it shown that the vanilla PINN exhibits failure modes when solving this equation. Here we use INN-TD solve the exact same problem.
0²
8²
ut u(x,)) q(x,))
(43)
Ox7
where the forcing function defined as:
q(x, v) - -aa1 m (asin(ar - ((o77)sin(arrc)ssn((aa) + &ssin(a
(44)
19

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
Table 9, Case 2: domain size 0, 12 (each case is repeated 10 times to obtain the statistics)
Model
Dim. (D)
colloc. points
# grid pts each dim
modes
# model purams.
Mean rel. L2 norm error w/ std.
Mean GPU wall time w/ std.
Mean GPU
VRAM usage std. (MB)
2
32
64
1.56 x 10
1 6.17
x
10 10-14
0.77 0.30
760 1 0
INN-TD
1.37
14
(4 sterations)
10
32
320
1.33 x 10
1
3.65
10
3.82
0.02
764 1:
2
32a
216
1.45 0.21
102.1 12.3
CP-PINN
epachs)
1,024
315.5 10.7
2
2,048
296
0.60 0.26
1,378 1. 30.7
1,014 1:
2
322
5
348
1,00 + 1.96 x 10
38.9 $ 3.05
866 0
KAN (50 steps)
128
5,498
4.81
2
128
20
28,578
0.53 = 0.03
369.6 = 2.06
11,008
10
se1, pe]
5m1.
s=4,
8.8 x 10
522, pa2
102, pad
Reladve L2 norm error
10"4
8.6 10
2,00 CCtev
Computation timee
8.4
10-5
2.03
x 10-1
8x 10'
10:4
5+1, p=]
1,38 C27exm
7.8x10 10
2-2, p-2
10
3,27 N.teew
6x10²
6x
10
Number of model paremeters
Number of model paremeters
Number of model para meters
Figure 12. Solving the 2D Poisson Eq. using INN-TD with different number of model parameters and hyperparameters and ==20 is used for all cases. The statistics are obtained by repeating each case times the shaded area represents standard deviation (a) convergence of INN-TD using different and p (b) statistics of total computational time (c) statisties of GPU VRAM usage
The solution domain is defined as: Ω E [-1, x [-1, The analytical solution to this problem is:
u( x2) sin πx) sin (a2my)
(45)
The comparison of INN-TD solution and the analytical solution shown in Fig. 13. can be seen that INN-TD achieves high accuracy without any fallure modes. Furthermore, study the convergence of INN-TD with different hyperparameters. The result is shown in Fig. 14. As expected, INN-TD shows the convergence property where larger number of model parameters leads to higher accuracy. Higher and can also decrease the error.
E.2.3. SPACE-TIME PDE EXAMPLE
In this example, we show the convergence property of INN-TD data-free solver for space-time problem. The governing PDE is shown below:
ú( 1) + kAu(a,, () = b( 1) in Qs 8 a
(46)
where & is heat conductivity. b(x, is defined as:
2 2 ( =o(()) + (v --o())))))
() = exp
(47)
22
where is the standard deviation that characterizes the width the heat source; (t), (t)) is the heat source center. This equation models the heat transfer with a moving heat source. The 4D space-time time continuum is adopted as: Qg = [-10, 10]3 S4 = (0, 0.1).
20

---

Interpolating Neural Network-Tene Decompasition (INN-TD)
INN-TD solution
Exact solution
Absolute error 1e-6
0.88
0.88
2.88
0.75
0.75
0.75
0.66
0.66
2.56
0.44
0.50
0.44
0.50
2.24
0.25
0.22
0.25
0.22
0.25
1.92
0.00
0.00
x
0.00
0.00
0.00
1..60
1.28
0.25
-0.22
-0.25
-0.22
-0.25
0.96
-0.44
-0.44
0.50
-0.50
-0.5
-0.6
-0.6
0.75
-0.75
-0.75
0.32
-0.88
-0.88
0.00
-0.5 0.0 0.5
-05
0.0 0.5
-0.5
0.0 0.5
X1
X1
X1
(a)
(b)
(c)
Figure 13. Solving the 2D Helmholtz Eq. using INN-TD 1, 20) with modes and 250 elements along each direction. The total solution time is 0.48 sec (a) INN-TD solution (b) exact solution (c) point-wise absolute error
10
6.2 10
9.4 10'
1_p11
s-4, p=1
22,p02
Reletive L2 norm eroo
10
2.06
9x
5.8x10
10-5
Timee Co
88x
5.
3.24
86x10 10
1.85
10-4
1
s=1, p=1
544, p=1
5-4 p=1
$2x 10
pm2
x2x10 10
10
583. pa3
5=3, pa3
10
60
10a
4 10'
6 10'
10"
4x
6x
102
Number of model parameters
Number model parameters
Number of model parameters
(c)
Figure 14. Solving the Helmholtz Eq. using INN-TD with different number of model parameters and hyperparameters s and p. a 20 is used for all cases. The statistics obtained by repeating each case for 10 times and the shaded area represents standard deviation (a) convergence of INN-TD TD using different s and p (b) statistics of total computational time (c) statistics of GPU VRAM usage
To investigate the convergence of INN-TD for space-time problems, we use the following forcing function:
b(x,t) t) = 2(1 - 2y²) - -e - (100t 55
5)3) - 150 e -0 -1000100-----00 + (1
+ 2(1 2(100t
(48)
1515)) 1200x + 1000 20000t) -0:000))e=02-0 -
15c 15t v -1000-----00---
As result, the exact solution be written
100))c-2((0-10000--00-0000000011 -2-1211000----
exact (xx,) =(1 -150)e -
(49)
sample snapshot of the solution is shown in Fig. 15 (a).
Since it's straightforward for numerical integration of C-HiDeNN interpolation function, here we define the relative 1.2
21

---

Interpolating Neural Network-Tenso Decomposition (INN-TD)
s=p=1
s=p=2
Relative LZ norm error
10
5=p=3
7.7e-01
0.7
0.65
0.6
10
0.55
0.5
0,45
Solutlon
0.4
0.35
10
0.3
0.25
0.2
0.15
10-4
0.0e+00
104
model parameters
(a)
Figure 15. Space-time PDE: (a) snapshot of the space-time solution (b) convergence for the 4D space-time problem: relationship between number model parameters and relative L2 norm error
norm error in the integral form:
( (waract(x (x, t) J Jux t))² dxdtt)
(50)
(ʃ (uezact(z)) dæddt
We investigated the convergence properties of INN-TD using different hyperparameters s and in C-HiDeNN interpolation. As shown in Fig. 15 (b), the error of INN-TD decreases more model parameters are utilized. This guaranteed convergence distinguishes INN-TD from other black-box deep learning models, where an increase model parameters does not necessarily result in reduced error. Additionally, the convergence rate can be controlled by adjusting s and p. providing flexibility in designing the INN-TD according to the accuracy requirement.
E.2.4. SPACE-PARAMETER-TO (S-P-T) PDE EXAMPLE
To solve the 6D S-P-T problem, used 101 grid points to discretize each input dimension of (z, y, 2, P,t). 100 modes were used for INN-TD approximation. Therefore, the INN-TD interpolation function can be written as:
100
Ju(x)
Σ
Ne(o)u!m) Ny(y)u!m) (N(xuu!")) (akkuu ("
(mm) Ne(t))(
(51)
Algorithm was used solve S-P-T problem, and took in total 155 to run the solver on a single NVIDIA RTX A6000 GPU.
Since the analytical solution is not available for this problem, used the finite element method (FEM) generate the simulation data and treated it as the ground truth. For ease of validation, the same spatial mesh size (100 x 100 x 100) and 100 time steps were adopted in FEM simulation. We repeated running FEM simulations with 10 randomly selected parametric input pairs (k: P). The relative L.2 norm error is computed as follows:
Σ [Julx, Ujj Zas kb, Pa. ta) - u FEM (I, Uj. Zas tai ky P.ja
(PEM(1;
Σ idja,deedd
FEM (z,, My. Zas tai Ko P.)
isjaabiedd
(52)
where i,J. a, b, e, d refer the data index of each input dimension.
22

---

Interpolating Neural Network-Tenson Decomposition (INN-TD)
E.2.5. INVERSE OPTIMIZATION PROBLEM
In this problem, we solved the S-P-T problem using INN-TD data-free solver. As a result, we obtained the S-P-T interpolation function shown in Eq. 51.
To test the effectiveness of INN-TD for the inverse optimization problem, we randomly sampled 100 different parametric input pairs (k,, i = 1,.., 100, and generated the corresponding ground truth through high-fidelity FEM simulation. Then we used Adam optimizer to optimize the loss function as shown in Eq. 13. The learning rate was chosen as 0.1. To ensure the parameters (kg, P.) are within the predefined range, we added box constraint on the parameter such that:
Kmin S lis kmaz
Pmin S P. < Pmax
For each optimization case, used uniform distributions for the initial guess. The mean and standard deviation for the relative L2 norm enror and parameter error were then computed across all cases.
E.2.6. VECTOR SOLUTION FIELD: ELASTICITY PROBLEM
In this example, INN-TD is used to solve the 3D elasticity problem in solid mechanics where the solution field is a vector field. The solution u. = [un (Ir, y, =), (2, y, v), uz(2, y. 2)) represents the displacement along the I, y and z directions. The governing equation is shown below:
jai + (μx + X)ujji + +b; =
(53)
where Einstein summation is used in the index notation, i = 1,2,3 and j 1, 2, X, i are the Lamé constants. The solution domain is defined Q [0, x [0, [0, 2] and homogeneous boundary condition is assumed. The forcing function b, is defined as:
b, 922765196.706387 nzr(x 1) sin(n)nnn))))))) sin 1im= 58745053. 12 cos(T22Tx)) (2y sin
Ramz
77745056.1097444 ny cos(n)Ty) sin (n ( sin(n1 sin (1)512)
b2
922765186.7063322 n² sT22 y(y - 5874550663.19974 71 1) cos (nirt) sin
587450558..00748 sin() (o3())) (z 523375701.269788-sim
sin (>
bg
738212149.36510 ná sinnnnnnnn (1 sin(n
293725281.554873 n (in (2y 1) cos 2232
2037252181554873 m2 - sin(n1 cos (>LT=) 261787850.634994 sin(ngTT)) (in
(54)
The analytical solution to the problem is:
(z, y: = 0.00097235487337877449 (x² sin(n1 sin
nime
2
u2( y. 0.0009723548737867799 (y² y) (in(T))T sin
22552
(55)
Ug u2(2, y. 0.0009723548733787749 (in(nggT2) (int:sstsin(n))))
The convergence of INN-TD shown in Fig. 16.
E.2.7. OPERATOR SOLVING
INN-TD can also be extended approximate PDE operators. As an example, we show that INN-TD can be leveraged approximate the PDE operator for the following equation:
Ou
0
Ou
k(x
f(x)
(56)
at
dr
with homogeneous initial and boundary conditions. Here we aim to approximate the PDE operator from the conductivity field k(x)) to the space-time PDE solution u(x, t).
23

---

Interpolating Neural Network-Tensor Decomposition (INN-TD)
1.E+00
Estimated error messurement based
convergence
FEM, slope:
Largest FEM
1.E-01
Grid points: 8.2e6, 2,452 sec
Energy norm error
20K 20K 40K nodcs Grid points: 1.7
1.E-02
INN-TD $1, p1, slope:
5K 10K nodes
1.E-03
Grid points: 2.7e11, sec
INN-TD s2,
FEM
INN-TD 1,
1.E-04
INN-TD s4, p1
INN-TD 2,pp
INN- -IN-TD a6, s6 p1
INN-TD
Grid points: 6.2c8, 5.1 sec
INN-TD < - 6,
1.E-05
1.E-06
1.E-05
1.E-04
1.E-03
1.E-02
1.E-01
mesh size (h)
Figure 16. Convergence for the elasticity problem: relationship between mesh size and energy
To account for the arbitrariness of k<x), we discretize and assume the nodal values of k(x) are subjected the following covariance function:
(x; Iz
C()), o² exp
(57)
212
where hyperparameters a and / refer to standard deviation and length scale, respectively. The covariance matrix can be further decomposed using cigen-decomposition and arbitrary k(x) is approximated using the following equation:
the
k(c, C) kp +
(Z
VAJonaCs
(58)
where ky is the mean value; A is the J-th eigenvalue; ΦI, refers to the I-th component of J-th eigenvector; C, is the J-th uncorrelated variable; (x) is the I-th C-HiDeNN basis function. As a result, the INN-TD approximation of the PDE operator can be written as:
Fu(x,C,t)= = Σ
u (m) (x)el (") (L)) (G)os"" (C2) .... (6lo) (Gn, )us
(59)
m==1
where each univariate function is approximated using C-HiDeNN basis functions. The details of the model discretizations shown in Table 10. INN-TD requires 96.74 + 4.65 see for the offline computation. Fig. 17 shows that INN-TD gives accurate predictions as compared to the corresponding finite difference solutions for different realizations of the conductivity field.
Table 10. Summary of the INN-TD model for approximating PDE operator
Discretization variable
Space (x)
Parameter for k (C)
Time (t)
Domain
[0. 1]
[-5, 5]
[0, 0.01]
Number of independent variables
71
1
Number of elements
101
151
Number of eigenvalues
71
24

---

Interpolating Neural Network-Tensee Decomposition (INN-TD)
(a)
Conductiv ty
INN-TD prediction
Finite difference solution
Pointwise 1e-5
11.0
0.008
0.008
3.2
0.007
0.007
2.8
0.008
0.008
0.008
10.5
0.006
0.006
2.4
0.006
0.005
0.006
0.005
0.006
2.0
k00
10.0
0.004
0.004
1.6
0.004
0.004
0.003
0.004
0.002
0.002
0.002
0.002
0.002
0.8
9.0
0.000
0.000
0.000
0.000
0.000
0.0
1.0
Conductiv ty
INN-TD prediction
Finite difference solution
Pointwise wi error
0.008
0.008
9.0
(b)
15.0
12.5
0.008
0.007
0.006
0.008
0.007
0.006
0.008
7.5
6.0
10.0
0.006
0.005
0.006
0.005
0.006
KD0
0.004
0.003
0.004
0.003
0.004
3.0
5.0
0.002
0.002
0.002
1.5
2.5
0.001
0.001
0.000
0.000
0.000
0.000
0.000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Figure 17. Results for operator solving task when conductivity field k(x)) is (a) sinusoidal function (b) random function.
25


Reviews:
[{"review_id": "NYN3SEj38F", "reviewer": "Reviewer_mqbf", "summary": "This paper introduces a new framework, Interpolation Neural Network-Tensor Decomposition (INN-TD), designed to efficiently and interpretably solve high-dimensional partial differential equations (PDEs) encountered in large-scale physics-based problems. The key innovation lies in integrating the local interpolation techniques of finite element methods with deep neural network architectures.  Additionally, by leveraging tensor decomposition, the method represents a multivariate function as a product of univariate functions, thereby reducing multidimensional integration to a series of one-dimensional integrations.   Experimental evaluations in data-driven training, data-free solving, and inverse optimization tasks demonstrate that INN-TD outperforms standard models like vanilla MLP, SIREN, KAN, and CP-PINN in terms of both accuracy and computational efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive suggestions, which can greatly improve the current manuscript.\n\n> Explanation of the interpretability\n\nAs stated by *Doshi-Velez et al. (arXiv:1702.08608)*, model interpretability refers to the ability to explain a model’s behavior in a way that is understandable to humans. INN-TD is an interpretable machine learning algorithm designed for scientific and engineering problems, offering transparency in its structure, clear convergence guarantees, and explainable insight into how data or parameter changes affect outcomes. (1) INN-TD’s learning parameters are interpretable nodal values of TD components, in contrast to the opaque weights and biases of neural networks. As shown in the newly added *[Fig 1](https://anonymous.4open.science/r/i7_/i0.png)*, INN-TD can be interpreted as a specially pruned MLP where the first 2 hidden layers represent linear finite element basis functions. (2) the algorithm has a well-defined convergence theory: as the number of segments (bins) and modes increases, the approximation error decreases. We added [Fig. 2](https://anonymous.4open.science/r/i7_/p1.png) to show its convergence. We can see that increasing the number of model parameters leads to a predictable error reduction at a rate of $p+1$. (3) INN-TD’s results are explainable—high or low accuracy in specific regions can be attributed to the quality of segmentation. A newly added [Fig. 3](https://anonymous.4open.science/r/i7_/i1.png) shows how we improve accuracy by refining the grid near the nonlinear region for the Poisson equation.\n\n\n> Theoretical claims\n\n It has been proved in *(Lu, Y, et. al Computational Mechanics 72, no. 2 (2023): 333-362.)* that the convergence rate of C-HiDeNN for solving PDE in terms of L2 norm error is $p+1$. It has also been proved that the INN-TD is a universal approximator and the following error bounds hold ($E$ refers to the energy norm):\n\n$\\|u^{\\text{C-HiDeNN}}-u^{\\text{exact}}\\|_E \\leq \\|u^{\\text{INN-TD}}-u^{\\text{exact}}\\|_E\\leq \\|u^{\\text{FEM}}-u^{\\text{exact}}\\|_E $\n\nWe have added this reference in the revision.\n\n> Running time statistics\n\nWe have rerun all the numerical examples. The statistics are shown in *[Table 1](https://anonymous.4open.science/r/i7_/t1.png)* and *[Table 2](https://anonymous.4open.science/r/i7_/t2.png)*.\n\n\n> Typo in equations\n\nThanks for pointing these out. We have carefully checked and revised all the equations. We will upload the revised manuscript after the discussion period.\n\n\n> Q1: missing hyperparameters in Eq 7.\n\nThe parameters $a,s,p$ still exist. We omitted them for simplicity of notation. We have added this clarification.\n\n> Q2: low dimensionality of solution\n\nWhen solving complex physical problems, standard numerical methods such as the finite element method (FEM) will result in a huge number of unknowns. In many cases, however, the problem solution lives in a subspace of a much smaller dimension. For example, the FEM solution to a 2D physical problem can be represented by a 2D matrix. Using PCA, this full matrix can be approximated using a few principal components in many cases. INN-TD can be treated as the generalization of PCA to general $N$-dimensional problems where the discretized solution can be deemed as a $N$-th order tensor. The decomposed TD components can then be approximated using the C-HiDeNN basis function as a locally supported basis function.\n\n> Q3: missing refs \n\nThanks for the suggestion! We have done a thorough literature review and added the reference in the manuscript as shown in *[Table 3](https://anonymous.4open.science/r/i7_/ref.png)*.\n\n> Q4: training/solving schemes\n\nAs listed in Algorithm 1-2, we have both boosting (multi-stage) and all-at-once (single-stage) algorithms for the training task. Boosting is flexible since one doesn't need to predefine the number of modes. However, given the same number of modes, the all-at-once algorithm has higher accuracy. The same rule applies to solving tasks. Although we only used the boosting algorithm in the current manuscript for the solving task, the all-at-once solving scheme is also available and is explained in *(J. Guo, et. al. arXiv:2503.13933)*. \n\n> Q5: choosing $s,p,a$\n\n We have thoroughly investigated the influence of hyperparameters in the new solving experiments as shown in *[Fig 3](https://anonymous.4open.science/r/i7_/p1.png)* and *[Fig 4](https://anonymous.4open.science/r/i7_/p2.png)*. As expected, larger $s$ and $p$ will lead to better accuracy. We also found that when $a>20$, the results remain the same. Furthermore, the interpretability of INN-TD allows us to choose $s,p$ for different domains thanks to the locally supported nature of the approximation as shown in the example of *[Fig 2](https://anonymous.4open.science/r/i7_/i1.png)*. \n\n> Q6: details of GPU wall time\n\nThe GPU wall time refers to the training/solving time. Once trained/solved, inference of INN-TD is very fast (~ ms).", "claims_and_evidence": "The authors make two claims in the paper. \n\nThe first claim is that the INN-TD method is more scalable. This is evident because it converts the original high-dimensional finite element interpolation problem into multiple one-dimensional function fitting problems. \n\nThe second claim is that INN-TD offers better interpretability. However, I find it hard to understand why INN-TD would be more interpretable compared to other PDE solvers like PINNs. In the paper, the authors only state, “C-HiDeNN is interpretable as it uses locally supported basis functions and the number of elements in C-HiDeNN is closely related to the resolution,” to justify the enhanced interpretability of INN-TD. This explanation is clearly insufficient and does not directly address the issue of interpretability.", "methods_and_evaluation_criteria": "This paper employs a variety of experiments to evaluate the performance of INN-TD, including data-free tasks, general space-parameter-time PDEs, and inverse optimization tasks. The evaluation results on these datasets provide strong support for the proposed method.", "theoretical_claims": "This paper does not provide any theoretical analysis or supporting proofs.", "experimental_designs_or_analyses": "Since the main claims of this paper focus on scalability and interpretability, I suggest that running time statistics be provided for each experiment.", "relation_to_broader_scientific_literature": "This paper falls under the category of AI for Science, or more specifically, Tensor Networks (TN) for Science, and therefore it has extensive connections with the literature on tensor networks, machine learning, and PINNs.", "essential_references_not_discussed": "Using tensor decomposition (not limited to CP) for solving high-dimensional PDEs is a very interesting research topic, there are some research works that have already proposed to solve the high-dimensional PDEs in different aspects. I strongly recommend that the authors expand their review in the related works section to properly acknowledge and respect prior contributions in this area.", "other_strengths_and_weaknesses": "**Strengths**\n\n(1) The paper employs the idea of multivariate function decomposition by decomposing a multivariate function into a tensor product of several univariate functions, which offers a clever and efficient way to solve high-dimensional physical problems.\n\t\n(2) The authors utilize FEA basis functions as the basis in the tensor function decomposition, enabling straightforward incorporation of constraints such as boundary conditions when solving PDEs.\n\n**Weaknesses**\n\n(1) The paper contains numerous notational errors, including in equations (2), (5), (8), (10), (13), and (18).\n\n(2) The use of tensor decomposition (not limited to CP) for solving high-dimensional PDEs has been widely studied in recent years. The authors should expand their review in the related works section to thoroughly cover this topic and properly acknowledge previous contributions.", "other_comments_or_suggestions": "None.", "questions_for_authors": "**Questions**\n\n(1) In equation (7), do the basis function parameters s, a, and p still exist? If they are omitted, please provide additional clarification.\n\n(2) Some physical systems may be low-dimensional, so why is the resulting multivariate function of low tensor rank?\n\n(3) Using tensor decomposition (not limited to CP) for solving high-dimensional PDEs is not novel; I strongly suggest that the authors expand the related works section on this topic to properly acknowledge prior work.\n\t\n(4) Why not solve for all the parameters at once, or why not train continuously in one phase instead of using a staged training approach? What benefits does the two-phase training method bring? I am curious about the motivation behind this training scheme and how the results would compare if only the first phase or only the second phase were used.\n\t\n(5) How are the FEA parameters set? Since s, p, and a significantly impact performance, this is critical; I recommend that the experimental section provides detailed information on their settings.\n\n(6) Does the GPU wall time include both training and inference, or is it solely the training time?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1f7IUqdUTW", "reviewer": "Reviewer_9f9s", "summary": "This paper proposes the Interpolation Neural Network-Tensor Decomposition (INN-TD), which relies on learnable locally supported interpolation functions for finite element methods and functional tensor decomposition for approximating high dimensional multivariate functions. The authors show that INN-TD outperforms prior methods for data-driven and data-free modeling, at a fraction of the computational cost.\n\n########## Update after rebuttal: I thank the authors for addressing my questions and concerns in detail. I have updated my score accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your kind comments, and they are very helpful to improve the manuscript.\n\n\n> GPU memory footprint in the tables is missing.\n \n We have added the GPU memory and computational costs for all cases solved by INN-TD, as shown in *[Table 1](https://anonymous.4open.science/r/i7_/t1.png)* and *[Table 2](https://anonymous.4open.science/r/i7_/t2.png)*. Since INN-TD leverages locally supported basis functions with tensor decomposition (TD), and we use the Galerkin method to solve the weak form of the PDE, the resulting systems of equations have a sparse structure and require minimal memory footprints. This is different from the collocation method used by many other machine learning PDE solvers. \n\n\n> Meaning of interpretability in Table 5 and experimental demonstration\n\nThe interpretability of INN-TD is reflected in four aspects: 1. As shown in the new *[Fig 1](https://anonymous.4open.science/r/i7_/i0.png)*, INN-TD can be interpreted as a pruned MLP where the first 2 hidden layers represent linear finite element basis functions. As a result, we can easily control the mesh size of locally supported basis functions such that a fine mesh will be used for solutions with high nonlinearity. 2. C-HiDeNN is constructed by adding the 3rd hidden layer with hyperparameters $s$ and $p$. Larger $s$ and higher $p$ lead to better accuracy and faster convergence. 3. Boundary conditions are easily satisfied due to the Kronecker delta property. 4. Instead of using collocation methods where convergence cannot be easily guaranteed, we use the Galerkin formulation, and the numerical results certify the convergence rate for different C-HiDeNN parameter $p$ (see *[Fig 2](https://anonymous.4open.science/r/i7_/p1.png)* and *[Fig 3](https://anonymous.4open.science/r/i7_/p2.png)*). For experimental demonstration, we have added a new numerical example. In this example, we solve a 2D Poisson equation where the main nonlinearity only exists in a small portion of the whole domain as shown in *[Fig 4](https://anonymous.4open.science/r/i7_/i1.png)*. We leverage the interpretability of INN-TD by only refining the grid close to the nonlinear solution area and using larger $s$ and $p$ to increase the smoothness of the approximation. As a result, the point-wise absolute error is $\\mathcal{O}(10^{-6})$. As a comparison, it's not straightforward to design an optimal MLP to reasonably approximate the solution to this problem.\n\n> Solve real-life large-scale systems\n\nWe highly agree with you that INN-TD should be used to solve real-world large-scale problems to demonstrate its advantages over standard numerical methods. We have leveraged INN-TD to solve a large-scale additive manufacturing problem, which will be added to the appendix. The main difficulty of this problem is the inherent mismatch of length scales and high precision requirement, which will make standard methods suffer from expensive computational costs. Details of the problem setup and results can be found in *[Fig 5](https://anonymous.4open.science/r/i7_/AM.png)* and (J. Guo, et. al. arXiv:2503.13933).\n\n> Details of properties of C-HiDeNN are missing\n\nWe have added a section in the appendix to explain the detailed properties of C-HiDeNN, as summarized in *[Fig 6](https://anonymous.4open.science/r/i7_/c.png)*.\n\n> Compare INN-TD with other methods for inverse problems.\n\nIn INN-TD, we first solve the parametric PDE $u(\\mathbf{x_s}, x_t, \\mathbf{x}_p)$ by considering parametric variables such PDE coefficients as inputs. Then this parametric solution is used as a fast prediction model to minimize the difference between the measurements and predictions. PINN solves the inverse parametric problem by directly adding an additional data loss to the loss function.\n\n> Figure 2 is not clear\n\nThanks for pointing this out. We have modified Figure 2 as attached in *[Fig 7](https://anonymous.4open.science/r/i7_/f2.png)*\n\n> Pass in PDE parameters as inputs during training\n\nFor data-driven training tasks, INN-TD can be used for either approximating the solution for constant PDE coefficients or treating PDE coefficients as extra inputs $\\mathbf{x}_p$. In the training example shown in Sec 4.1.1, the input vector is $(x,y,k,P,t)$, which covers location, PDE coefficients, and time. \n    \n> Definition of norms\n\nWe used relative L2 norm error in a discrete sense since we want to have a fair comparison against collocation-based neural solvers, of which the error is computed on discrete points. But we can also easily compute the norms as integrals to examine the error. In Appendix D2.2 Eq. 44, we defined the error norm as the integral.\n\n> Meaning of $A^e$ and $A^i_s$\n\nWe have added a more detailed new *[figure](https://anonymous.4open.science/r/i7_/c0.png)* to explain the notations in C-HiDeNN basis function. $A^e$ refers to nodes within element $e$; $A^i_s$ is patch domain at node $i$ with patch size $s$. We will upload the updated manuscript after the discussion period.", "claims_and_evidence": "The authors’ main claims are that the INN-TD is faster, more scalable, and more interpretable than existing methods. They also claim that it leads to better performance and reduced memory footprint. Most of these claims are addressed in the results. The tables in section 4 show that INN-TD achieves lower error rates and is in general faster and more scalable to higher dimensional problems than alternatives. \n\nHowever, the authors do not show the GPU memory footprint in these tables, and I do not see any empirical demonstration of the claimed interpretability of INN-TD. Given that the authors describe C-HiDeNN as “fully interpretable” in Table 5, an experimental demonstration of this would be quite useful.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are mostly aligned with the application at hand. Given that the authors claim that INN-TD is applicable to large-scale systems, it would be interesting to also see a comparison of INN-TD to a system of greater real-world interest.", "theoretical_claims": "Given that C-HiDeNN is not necessarily well known by the entire community, I think it would be useful for the authors to include a more detailed description (maybe the results but not the proofs) of the properties of C-HiDeNN: locality, Kronecker delta property, partition of unity, etc.", "experimental_designs_or_analyses": "See note above in “Methods and Evaluation Criteria.” Also, it would be interesting to compare INN-TD with other methods in the inverse optimization task.", "relation_to_broader_scientific_literature": "This paper presents INN-TD, a method that appears more principled and grounded than prior black-box models for solving PDEs using neural networks. It is largely based on C-HiDeNN, but it also incorporates tensor decomposition for better scalability to high dimensional systems. The authors compare with prior works that have been applied to these tasks, notably KANs and PINNs, which is quite useful.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "Figure 2 is not very clear: the legend and axis labels are small, and the caption is quite vague.", "questions_for_authors": "1. For data-driven training, is the goal to solve one instance of a PDE (if the data pairs are $(x_i, y_i)$)? Is it also possible to pass in PDE parameters as inputs? Did the authors try this setup as well?\n2. Why are the norms (such as the one in equation 13) often finite-dimensional when dealing with the infinite-dimensional $u^*$ function?\n3. What are $A^e$ and $A^i_s$? I’m having a bit of trouble following the notation in Appendix A.\n4. What does “Fully interpretable” in Table 5 mean? I suggest the authors elaborate.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCE042WnFS", "reviewer": "Reviewer_tVvM", "summary": "This paper presents Interpolation Neural Network-Tensor Decomposition (INN-TD), a framework combining neural network approximation of PDE solutions with tensor decomposition methods. By incorporating locally supported interpolation functions, the author claims that INN-TD enhances accuracy, speeds up training and solving, and reduces memory usage. The authors validates the claim by performing the experiments on high-dimensional parametric PDEs in training, solving, and inverse optimization tasks.\n\n## update after rebuttal\n\nI have updated my score based on the author's reply to my concerns. While I'm nearly satisfied with the quality of the research, I strongly hope that the interpretability in the original manuscript is updated based on my rebuttal discussion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's valuable comments.\n\n> Running statistics and comparison\n\nWe have added the statistics for all training examples in *[Table 1](https://anonymous.4open.science/r/i7_/t0.png)*. The statistics for the solving examples are summarized in *[Table 2](https://anonymous.4open.science/r/i7_/t1.png)* and *[Table 3](https://anonymous.4open.science/r/i7_/t2.png)*. For the solving tasks, it's important to mention that INN-TD is not based on the collocation method, of which convergence cannot be generally guaranteed. Instead, INN-TD leverages the merits from the finite element method (FEM) and uses the Galerkin weak form to compute the PDE loss in the integral form. The resulting systems of equations are sparse and can be solved using matrix solvers instead of relying on stochastic optimization. Thus, INN-TD has guaranteed convergence when solving PDEs and much faster solving speed compared to others (even if the number of parameters is larger than other models). The statistics of error, computation time, and GPU VRAM usage are plotted in *[Fig 1](https://anonymous.4open.science/r/i7_/p1.png)*. It can be seen there is no fluctuation in the error, although the initial guess of the solution is initialized randomly using normal distributions. \n\n> Only scalar PDE examples\n\nWe have added 1 example of solving the elasticity equation in solid mechanics, which is a vector PDE. The results are in newly added *[Fig 2](https://anonymous.4open.science/r/i7_/e1.png)*, where we can control the accuracy of the model by choosing different $s$ and solve large-scale problems FEM cannot handle. Solving NS equations is the next step. It's worth noting that NS equations have been solved using tensor decomposition-based methods such as in *(Fernandes, Jeferson et al, CMAME 386(7):114102)*. Consequently, we are confident that INN-TD can be effectively extended to solve NS equations.\n\n> Discussion on the interpretability\n\nModel interpretability generally refers to the ability to explain a model’s behavior in a way that is understandable to humans. INN-TD is an interpretable machine learning algorithm with a transparent structure, clear convergence guarantees, and explainability of outcomes. (1) INN-TD’s learning parameters are interpretable TD components of nodal values in each mode. (2) INN-TD has a well-defined convergence theory: as the number of elements and modes increases, the error decreases. We added [Fig. 3](https://anonymous.4open.science/r/i7_/p1.png) to show its convergence behavior. (3) INN-TD’s results are explainable—high or low accuracy in specific regions can be attributed to the quality of segmentation. A newly added [Fig. 2](https://anonymous.4open.science/r/i7_/i1.png) shows how we improve accuracy by refining the grid near the nonlinear solution region when solving the Poisson equation.\n\n> Train/validation/test split and generalization for training tasks\n\nFor the training task, We sampled two parameters $(k, P)$ six times each, generating 36 time-dependent simulations. Seven cases (20%) were randomly chosen as the test set, including unseen parameter combinations. The remaining 29 cases were shuffled across space $(x, y)$, time $(t)$, and parameters $(k, P)$, then split into training and validation sets. This setup ensures the test set represents an out-of-domain distribution relative to the model’s exposure.\n\n\n> Performance of INN-TD for cases where PINN fails\n\nFrom *(Wang+ (2021), SIAM J. Sci. Comput., 43 (5): A3055.)*, the vanilla PINN exhibits failure modes when solving Helmholtz equation. Here we use INN-TD to solve the exact same problem. The results are shown in newly added *[Fig 4](https://anonymous.4open.science/r/i7_/h1.png)*. INN-TD can solve this equation very fast and accurately with guaranteed convergence rates. Moreover, we have also added these essential references as suggested by the reviewer.\n\n> Operator learning using INN-TD\n\nThanks for the suggestion! We have added a new example of using INN-TD to approximate PDE operators. The goal is to approximate a PDE operator without generating any data. Details can be found in newly added *[Fig 5](https://anonymous.4open.science/r/i7_/o1.png)*, where we successfully approximate the PDE operator mapping from the conductivity field $k(x)$ to the solution $u(x,t)$. We use Karhunen-Loève(KL) expansion together with C-HiDeNN basis functions to approximate arbitrary function $k(x)$. INN-TD can solve the problem very fast thanks to the benefits of TD for high-dimensional problems. We have included this problem in the appendix.\n\n> Large standard deviation in Table 4 \n\nWe have tried 200 more runs, and the standard deviation has significantly decreased, as shown in the updated *[table](https://anonymous.4open.science/r/i7_/t.png)*.\n    \n> Definition of the color and axes in Fig 9.\n\nThanks for the question! The contour plot is the solution field. We have updated the figure as shown in *[Fig 6](https://anonymous.4open.science/r/i7_/s.png)*.", "claims_and_evidence": "The claims presented in the paper are largely supported by the experiments and theoretical considerations. However, I believe that additional efforts are necessary to fully validate them for the following reasons:\n\n1. In all the experiments, training was conducted only once. As a result, the reported strong performance could be attributed to favorable initial weights and a fortunate stochastic optimization process. It is strongly recommended to conduct multiple trials (e.g., three to ten) and report the average performance along with the standard deviation to assess statistical robustness.\n\n2. The experiments were conducted solely on a scalar equation, without considering real-world PDE systems such as the Navier-Stokes (NS) equations. This is a critical limitation, as Physics-Informed Neural Networks (PINNs) often perform well on simple scalar PDEs but struggle with NS turbulence. Evaluating the proposed method on a complex real-world PDE would strengthen its validity.\n\n3. Although interpretability is highlighted as a key advantage of the proposed method, no substantive discussion is provided on this aspect. Specifically, it remains unclear how the authors would interpret cases where INN-TD fails to approximate PDE solutions. Simply attributing failure to the optimization process of the interpolation neural network does not offer meaningful insights from an interpretability standpoint. A more detailed discussion on this issue is necessary.", "methods_and_evaluation_criteria": "The Benchmark dataset can include more challenging tasks, such as compressible NS equations (PDEBench) and various hydrodynamic simulations (The Well). \n\t\n[1] Takamoto, Makoto, et al. \"Pdebench: An extensive benchmark for scientific machine learning.\" Advances in Neural Information Processing Systems 35 (2022): 1596-1611.\n\t\n[2] Ohana, Ruben, et al. \"The well: a large-scale collection of diverse physics simulations for machine learning.\" Advances in Neural Information Processing Systems 37 (2024): 44989-45037.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "1. As mentioned above, it is essential to report the statistically averaged performance along with the standard deviation to ensure robustness.\n\n2. The authors should provide more details regarding the train/validation/test dataset split. Specifically, is the test set drawn from an in-domain distribution or an out-of-domain distribution? Clarifying this distinction is important for evaluating the generalizability of the model.\n\n3. In Tables 2 and 3, it would be preferable to maintain consistent conditions, such as the number of model parameters, to facilitate a fair comparison of performance. In the current version, the imbalance in experimental conditions makes it difficult to determine whether INN-TD genuinely outperforms other methods.\n\n4. The paper does not discuss the well-known failure modes of PINNs, as reported in [3,4]. To demonstrate the practical usefulness of the proposed method, it would be valuable to assess whether INN-TD can mitigate these optimization challenges without requiring special modifications, as extensively discussed in the PINN literature.\n\n[3] Wang+ (2021), SIAM J. Sci. Comput., 43 (5): A3055.\n\n[4] Wang+ (2022), arXiv:2203.07404.", "relation_to_broader_scientific_literature": "The proposed method can be applied any existing PINN method. Potentially, it could also be used as a neural operators, though not fully explored in the paper.", "essential_references_not_discussed": "As the references discussing PINN optimization issues, [3], [4] are missed ([4] is cited but no mentions on the optimization issue). \n\n[3] Wang+ (2021), SIAM J. Sci. Comput., 43 (5): A3055.\n\n[4] Wang+ (2022), arXiv:2203.07404.", "other_strengths_and_weaknesses": "Strength: \n \n* As long as shown in the paper's experiments, INN-TD allows simulating PDE solution much better accuracy than the other existing NN functions, with much shorter inference time. \n    \t \nWeakness: \n\n* Its extrapolation ability in out-of-distribution data points is unknown.", "other_comments_or_suggestions": "The standard deviation in Table 4 is very large. The number of the trials should be increased to reduce the fluctuation.", "questions_for_authors": "What is the definition of the color and axes of the object in Figure 9?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new framework, Interpolation Neural Network-Tensor Decomposition (INN-TD), designed to efficiently and interpretably solve high-dimensional partial differential equations (PDEs) encountered in large-scale physics-based problems.  The authors' rebuttal has successfully addressed some concerns about experiments and presentation clarity. These revisions should be incorporated into the final version.
Decision: Accept (poster)