SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also conducted some experiments to explore the use of L-BFGS when training SPINN. We found that training with Adam first and then fine-tuning with L-BFGS showed a slight increase in accuracy. Note that this training strategy is used by other works [7, 14] and is known to be effective in some cases. Tab. 2 shows the numerical results on three 3-d PDEs. Understanding the effect of the optimization algorithm is still an open question in PINNs, we believe that investigating this issue in the context of SPINN would be a valuable direction for future study.
Table 2: Numerical result of 3-d PDEs with L-BFGS fine-tuning. The number of training collocation points is  $64^{3}$  and a single outer loop L-BFGS is applied.
Diffusion	Helmholtz	Klein-Gordon
Adam	0.0041	0.0360	0.0013
Adam + L-BFGS	0.0041	0.0308	0.0010
Table 3: Full results of diffusion equation.  $N_{c}$  is the number of collocation points.
model	Nc	relative L2 error	RMSE	runtime (ms/iter.)	memory (MB)
PINN	163	0.0095	0.00082	3.98	1,022
323	0.0082	0.00071	12.82	2,942
643	0.0081	0.00070	95.22	18,122
PINN + modified MLP	163	0.0048	0.00042	14.94	1,918
323	0.0043	0.00037	29.91	4,990
543	0.0041	0.00036	134.64	22,248
SPINN	163	0.0447	0.00387	1.45	766
323	0.0115	0.00100	1.76	766
643	0.0075	0.00065	1.90	766
1283	0.0061	0.00053	2.09	894
2563	0.0061	0.00053	10.54	2,174
SPINN + modified MLP	163	0.0390	0.00338	2.17	766
323	0.0067	0.00058	2.44	768
643	0.0041	0.00036	2.59	768
1283	0.0036	0.00031	3.06	896
2563	0.0036	0.00031	12.13	2,176
Table 4: Full results of the Helmholtz equation.  $N_{c}$  is the number of collocation points.
model	Nc	relative L2 error	RMSE	runtime (ms/iter.)	memory (MB)
PINN	163	0.9819	0.3420	4.84	2,810
323	0.9757	0.3398	14.84	2,938
643	0.9723	0.3386	110.23	18,118
PINN + modified MLP	163	0.4770	0.1661	18.32	7,034
323	0.5176	0.1803	35.02	9,082
543	0.4770	0.1661	159.90	22,244
SPINN	163	0.1177	0.0410	1.54	762
323	0.0809	0.0282	1.71	762
643	0.0592	0.0206	1.85	762
1283	0.0449	0.0156	1.89	762
2563	0.0435	0.0151	3.84	1,146
SPINN + modified MLP	163	0.1161	0.0404	2.24	764
323	0.0595	0.0207	2.50	764
643	0.0360	0.0125	2.57	764
1283	0.0300	0.0104	2.76	764
2563	0.0311	0.0108	5.50	1,148
Table 5: Full results of the (2+1)-d Klein-Gordon equation.  $N_{c}$  is the number of collocation points.
model	Nc	relative L2 error	RMSE	runtime (ms/iter.)	memory (MB)
PINN	163	0.0343	0.0218	4.70	2,810
323	0.0281	0.0178	14.95	2,938
643	0.0299	0.0190	112.00	18,118
PINN + modified MLP	163	0.0158	0.0100	17.87	7,036
323	0.0185	0.0118	34.61	9,082
543	0.0163	0.0104	159.20	22,246
SPINN	163	0.0193	0.0123	1.55	762
323	0.0060	0.0038	1.71	762
643	0.0045	0.0029	1.82	762
1283	0.0040	0.0025	1.85	890
2563	0.0039	0.0025	3.98	1,658
SPINN + modified MLP	163	0.0062	0.0039	2.20	764
323	0.0020	0.0013	2.41	764
643	0.0013	0.0008	2.57	764
1283	0.0008	0.0005	2.79	892
2563	0.0009	0.0006	5.61	1,660
Table 6: Full results of the (3+1)-d Klein-Gordon equation.  $N_{c}$  is the number of collocation points.
model	Nc	relative L2 error	RMSE	runtime (ms/iter.)	memory (MB)
PINN	164	0.0129	0.0096	43.51	5,246
234	0.0121	0.0090	154.24	22,244
PINN + modified MLP	164	0.0061	0.0045	100.06	17,534
184	0.0059	0.0044	174.00	22,246
SPINN	164	0.0122	0.0091	2.45	890
324	0.0095	0.0071	2.98	892
644	0.0093	0.0069	9.22	2,172
SPINN + modified MLP	164	0.0064	0.0048	3.48	892
324	0.0022	0.0016	3.66	892
644	0.0012	0.0009	10.96	2,172
PNNN
sno NNns
(a) Training time: 5 sec
(b) Training time: 30 sec
(s) NNI ds
(c) Training time: 1 min
Rereee
$t = 0.25$
$t = 0.5$
$t = 1.0$
Figure 3: Visualized solution of nonlinear diffusion equation obtained by the baseline PINN and SPINN, both trained on  $64^{3}$  collocation points.
SPINN (Ours)
Absolute error (SPINN)
PINN
Absolute error (PINN)
(a) Training time: 5 sec
Figure 4: Visualized solution of Helmholtz equation obtained by the baseline PINN and SPINN, both trained on  $64^{3}$  collocation points.
PINN
Absolute error (PINN)
(b) Training time: 30 sec
PINN
Absolute error (PINN)
(c) Training time: 1 min
Figure 5: Visualized vorticity maps of  $(2 + 1)$ -d Navier-Stokes equation experiment predicted by SPINN. Three snapshots at timestamps  $t = 0.0, 0.5, 1.0$  are presented.
(d) Prediction (vorticity)
Figure 6: Visualized solution of (3+1)-d Navier-Stokes equation obtained by SPINN, trained on  $32^{4}$  collocation points. Each arrow is colored by its zenith angle.