SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A.1: Error of OSC and four interpolation methods on different PDEs problems:  $u(x) = x^4 - 2x^3 + 1.16x^2 - 0.16x$  (1-D linear),  $u(x) = \sin(3\pi x)$  (1-D non-linear),  $u(x,y) = x^2y^2 - x^2y - xy^2 + xy$  (2-D linear),  $u(x,y) = \sin(3\pi x)\sin(3\pi y)$  (2-D non-linear).
MODEL	1-D LINEAR	1-D NON-LINEAR	2-D LINEAR	2-D NON-LINEAR
NEAREST INTERPOLATION	2.3670×10-6	1.7558×10-2	1.9882×10-3	3.8695×10-2
LINEAR INTERPOLATION	1.8928×10-7	8.7731×10-4	3.4317×10-4	1.1934×10-2
CUBIC INTERPOLATION	3.5232×10-12	2.2654×10-7	2.9117×10-4	4.5441×10-3
OSC	3.4153×10-31	4.1948×10-8	1.7239×10-32	3.4462×10-5
B.1 Models and implementation details
In this section we introduce additional experimental details.
Model Specific details of model components are introduced below:
Component	Description	Configuration
MGN encoder	three-layer MLP	hidden size= 64
MGN processor	in total 3 processors, each three-layer MLP	hidden size= 64
MGN decoder	three-layer MLP	hidden size= 64
All MLPs have  $\mathrm{ReLU}:x\to \max (0,x)$  nonlinearities between layers.
Specific details of applying the OSC method are introduced as follows:
Table B.1: Model Components and Configurations.
Configuration	Time-oriented OSC	Space-oriented OSC
Polynomial Order	3	3
Collocation Points	2	2
Table B.2: OSC Configurations.
We note that the number of collocation points in one partition is the same for each spatial dimension; features also share collocation points (i.e., wind speeds and temperature measurements share the same partition).
Regarding the choice of collocation points, for regular boundary problems, we initialize collocation points using Gauss-Legendre quadrature, following the recommendation by Bialecki (1998)<sup>7</sup>. This choice is particularly apt for smooth and well-behaved functions over the integration interval, making it suitable for achieving accurate results without oscillatory behavior. In irregular boundary scenarios, we employ mesh points as collocation points. Importantly, the space-oriented OSC technique enables us to obtain values at any spatial position, rendering the entire space available for collocation points, which makes the adaptive collocation points possible.
Training A batch size of 32 was used for all experiments, and the models were trained for up to 5000 epochs with early stopping. We used the Adam optimizer (Kingma and Ba, 2014) with an initial learning rate of 0.001 and a step scheduler with a 0.85 decay rate every 500 epoch. For all datasets, we used the split of  $5:1:1$  for training, validating, and testing for a fair comparison. We set the parameter  $\beta$  for adaptive OSC as constant. In particular, we set  $\beta = \frac{1}{2} \times \min$  partitions width. This ensures that collocations will not shift by more than half of the partition range in one adaptation step, lending stability and robustness to the adaptive mesh.
Space and Time Resolutions We additionally add a more detailed overview of the experiments regarding data setup, especially for space and time resolution, such as total rollout time, space, and time resolution for both OSC space and time in Table B.3.
Note that the rollout time and the other times  $\Delta t$  refer to the original dataset prediction horizons. For instance, in the Black Sea dataset, the ground truth data is provided every day (1 d), while the prediction horizon consists of 10 steps (i.e., a total of 10 days). For datasets including more than a single scalar value (Ocean Currents and Black Sea have 3 features, while 3D Navier-Stokes have 5), we the OSC is applied feature-wise.
Note that other choices may be made. For instance, Chebyshev grids may offer advantages such as spectral convergence properties and the ability to handle function singularities more effectively. This can be an interesting avenue for future research.
Experiment	Total Rollout Time	Step (Δt)	Resolution
Ground Truth	Collocation	Ground Truth	Collocation Points
Heat Equation	5 s	0.1 s	0.2 s	64 × 64	12 × 12
Damped Wave	10 s	1 s	2 s	64 × 64	32 × 32
2D Navier-Stokes	10 s	1 s	2 s	64 × 64	32 × 32
Ocean Currents	10 h	1 h	2 h	73 × 73	36 × 36
Black Sea	10 d	1 d	2 d	5000 nodes	1000 nodes
3D Navier-Stokes	0.5 s	0.05 s	0.1 s	262,144 nodes	32,768 nodes
Table B.3: Experiment details and parameters for each dataset.
Example Rollout To further elucidate the rollout procedure with GRAPHSPLINENETS, we showcase a simple example. For the 2D Navier-Stokes experiment, the ground truth trajectories consist of 10-time steps (10 seconds), with a step size of  $\Delta t = 1$  second. In our model, the neural network autoregressively performs the rollout 5 times with  $\Delta t = 2$  seconds, and the OSC (Orthogonal Spline Collocation) method is used to interpolate through the skipped steps. In other words, the ground truth consists of
$$
Y _ {t}, \quad t \in \{1, 2, 3, 4, 5, 6, 7, 8, 9, 1 0 \},
$$
while the neural networks in our model provide the interpolated outputs as
$$
\tilde {Y} _ {t _ {\text {i n t e r p o l a t e}}}, \quad t _ {\text {i n t e r p o l a t e}} \in \{2, 4, 6, 8, 1 0 \},
$$
which are then further refined using the Time-Oriented OSC to produce
$$
\hat {Y} _ {t} = \text {T i m e O r i e n t e d O S C} \left(\tilde {Y} _ {t _ {\text {i n t e r p o l a t e}}}\right).
$$
B.2 Heat Equation
The heat equation describes the diffusive process of heat conveyance and can be defined by
$$
\frac {\partial u}{\partial t} = \Delta u \tag {16}
$$
where  $u$  denotes the solution to the equation, and  $\Delta$  is the Laplacian operator over the domain. In a  $n$ -dimensional space, it can be written as:
$$
\Delta u = \sum_ {i = 1} ^ {n} \frac {\partial^ {2} u}{\partial x _ {i} ^ {2}} \tag {17}
$$
Dataset generation We employ FEniCS (Logg et al., 2012) to generate a mesh from the domain and solve the heat equation on these points. The graph neural network then uses the mesh for training.
B.3 Wave Equation
The damped wave equation can be defined by
$$
\frac {\partial^ {2} w}{\partial t ^ {2}} + k \frac {\partial w}{\partial t} - c ^ {2} \Delta w = 0
$$
where  $c$  is the wave speed and  $k$  is the damping coefficient. The state is  $X = (w, \frac{\partial w}{\partial t})$ .
Data generation We consider a spatial domain  $\Omega$  represented as a  $64\times 64$  grid and discretize the Laplacian operator.  $\Delta$  is implemented using a  $5\times 5$  discrete Laplace operator in simulation; null Neumann boundary condition are imposed for generation. We set  $c = 330$  and  $k = 50$  similarly to the original implementation in Yin et al. (2021).
B.4 2D Incompressible Navier-Stokes
The Navier-Stokes equations describe the dynamics of incompressible flows with a 2-dimensional PDE. They can be described in vorticity form as:
$$
\frac {\partial w}{\partial t} = - v \nabla w + \nu \Delta w + f
$$
$$
\nabla v = 0 \tag {18}
$$
$$
\boldsymbol {w} = \nabla \times \boldsymbol {v}
$$
where  $v$  is the velocity field and  $w$  is the vorticity,  $\nu$  is the viscosity and  $f$  is a forcing term. The domain is subject to periodic boundary conditions.
Data generation We generate trajectories with a temporal resolution of  $\Delta t = 1$  and a time horizon of  $t = 10$ . We use similar settings as in Yin et al. (2021) and Kirchmeyer et al. (2022): the space is discretized on a  $64 \times 64$  grid and we set  $f(x,y) = 0.1(\sin(2\pi(x + y)) + \cos(2\pi(x + y)))$ , where  $x, y$  are coordinates on the discretized domain. We use a viscosity value  $\nu = 10^{-3}$ .
B.5 Ocean Currents
The ocean currents dataset (Nardelli et al., 2013; Marullo et al., 2014) is composed of hourly data of ocean currents with an Earth grid horizontal resolution of  $1/12$  degrees with regular longitude/latitude equirectangular projection<sup>8</sup>. We employ data from 2019-01-01 to 2022-01-01 at a depth of 0.49 meters. We consider an area in the Pacific ocean characterized by turbulent currents corresponding to latitude and longitude  $(-126 \sim -120, -36 \sim -30)$ . We use currents eastward and northward sea water velocities as variables, i.e., uo and vo, respectively as well as water temperature to.
B.6 Black Sea
The Black Sea dataset is composed of daily real-world measurements of ocean currents and temperatures (Ciliberti et al., 2021). The resolution of the raw data is  $1/27^{\circ} \times 1/36^{\circ}$ . We employ data starting from 01/01/2012 until 01/01/2020; we split training, validation and testing with ratios of 5:1:1 as in the other datasets in the paper sequentially on the temporal axis; i.e., so that the model has not seen data from 2018 or 2019 during training. Of the data points in the grids, less than  $50\%$  actually cover the Black Sea due to the irregular shape of the sea. To obtain a mesh, we subsample the grid using Delaunay triangulation to 5000 nodes for baselines and 1000 nodes for GRAPHSPLINENETS, which results in a non-uniform mesh with an irregularly-shaped boundary akin to Lienen and Gunnemann (2022). We can observe a detail of the ground truth, non-adaptive and adaptive meshing alongside GRAPHSPLINENETS errors in Figs. B.1 to B.3 respectively. We use currents eastward and northward sea water velocities as variables, i.e., uo and vo, respectively, as well as water temperature to at a single depth of  $12.54\mathrm{m}$ . We normalize features by the mean and standard deviation of the training samples. The dataset is available to download from the Copernicus Institute<sup>9</sup>.
B.7 3D Compressible Navier-Stokes Equations
These equations describe the dynamics of compressible fluid flows (Takamoto et al., 2022):
$$
\partial_ {t} \rho + \nabla \cdot (\rho \mathbf {v}) = 0, \tag {19a}
$$
$$
\rho \left(\partial_ {t} \mathbf {v} + \mathbf {v} \cdot \nabla \mathbf {v}\right) = - \nabla p + \eta \triangle \mathbf {v} + (\zeta + \eta / 3) \nabla (\nabla \cdot \mathbf {v}), \tag {19b}
$$
$$
\partial_ {t} \left[ \epsilon + \frac {\rho v ^ {2}}{2} \right] + \nabla \cdot \left[ \left(\epsilon + p + \frac {\rho v ^ {2}}{2}\right) \mathbf {v} - \mathbf {v} \cdot \sigma^ {\prime} \right] = 0, \tag {19c}
$$
where  $\rho$  is the mass density,  $\mathbf{v}$  is the velocity,  $p$  is the gas pressure,  $\epsilon = p / (\Gamma -1)$  is the internal energy,  $\Gamma = 5 / 3$ ,  $\sigma^{\prime}$  is the viscous stress tensor,  $\eta$  the shear and  $\zeta$  the bulk viscosities. We employ a downsampled version of the dataset from PDEBench of Takamoto et al. (2022), originally  $128\times 128\times 128$  and subsampled to  $64\times 64\times 64$  up to  $N_{t} = 10$  for the ground truth data, while the
<sup>8</sup>Available at https://data.marine.copernicus.eu/product/GLOBAL_ANALYSISFORECAST_PHY_001_024
${}^{9}$  Available at https://data.marine.copernicus.eu/product/BLKSEA MultiYEAR PHY_007_004/description
Wind Speed Ground Truth
Figure B.1: Wind speed ground truth for the Black Sea dataset.
Without Adaptive Collocation Points
Figure B.2: MSE for GRAPHSPLINENETS predictions on the Black Sea dataset without adaptive meshing.
With Adaptive Collocation Points
Figure B.3: MSE for GRAPHSPLINENETS predictions with adaptive meshing. Adaptive meshing allows for capturing more dynamic regions better, as evidenced by the lower worst error.
collocation points lay on a dynamic grid of  $32 \times 32 \times 32 = 32768$  nodes. We use a total of 600 trajectories which we divide into training and testing with a  $90\%$  split as done in PDEBench.
B.8 Hardware and Software
Hardware Experiments were carried out on a machine equipped with an INTEL CORE I9 7900X CPU with 20 threads and a NVIDIA RTX A5000 graphic card with 24 GB of VRAM.
Software Software-wise, we used FEniCS (Logg et al., 2012) for Finite Element simulations for the heat equation experiments and PyTorch (Paszke et al., 2019) for simulating the damped wave and Navier-Stokes equations. We employed the Deep Graph Library (DGL) (Wang et al., 2020) for graph neural networks. We employ as a main template for our codebase the Lightning-Hydra-Template which we believe is a solid starting point for reproducible deep learning even outside of supervised learning (Berto et al., 2023); the Lightning-Hydra-Template combines the PyTorch Lightning library (Falcon et al., 2019) for efficiency with modular Hydra configurations (Yadan, 2019).
$^{10}$ https://github.com/ashleve/lightning-hydra-template
Figure C.2: Visualization of the Navier Stokes dataset.
Figure C.3: Wave dataset prediction results. GRAPHSPLINENETS generates more stable and smoother predictions compared to baselines.
Figure C.4: Ocean dataset prediction results. GRAPHSPLINENETS generates more stable and smoother predictions.
Figure C.5: Ocean dataset prediction error. GRAPHSPLINENETS contains the error better than baselines.