Scalable Mechanistic Neural Networks
Authors: Jiale Chen
Dingling Yao
Adeel Pervez
Dan Alistarh
Francesco Locatello
Venue: ICLR 2025 Poster (2025)
Keywords: Scientific Machine Learning
Ordinary Differential Equations
Time Series
Dynamical Systems
Primary area: learning on time series and dynamical systems
TL;DR: We introduce a scalable solver for the mechanistic neural networks that reduces time and memory costs from cubic/quadratic to linear in the sequence length, enabling long-horizon scientific modeling of dynamical systems in the neural networks.

Abstract: We propose Scalable Mechanistic Neural Network (S-MNN), an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) (Pervez et al., 2024), we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. This significant improvement enables efficient modeling of long-term dynamics without sacrificing accuracy or interpretability. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources. Consequently, S-MNN can drop-in replace the original MNN in applications, providing a practical and efficient tool for integrating mechanistic bottlenecks into neural network models of complex dynamical systems. Source code is available at https://github.com/IST-DASLab/ScalableMNN.

Published as a conference paper at ICLR 2025
SCALABLE MECHANISTIC NEURAL NETWORKS
Jiale Chen, Dingling Yao, Adeel Pervez, Dan Alistarh, Francesco Locatello Institute of Science and Technology Austria (ISTA) jiale. chen@ist.ac. act
ABSTRACT
We propose Scalable Mechanistic Neural Network (S-MNN), an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) (Pervez et al., 2024), we reduce the computational time and space complexities from cubic and quadratic with respect the sequence length. respectively, to linear. This significant improvement enables efficient modeling of long-term dynamics without sacrificing accuracy or interpretability. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources. Consequently, S-MNN can drop-in replace the original MNN in applications, providing a practical and efficient tool for integrating mechanistic bottlenecks into neural network models of complex dynamical systems. Source code is available at attps://git commcccm/////// IST- SSST-DASLab// MaAASaaa/SaaaaleMMMW
INTRODUCTION
The Mechanistic Neural Network (MNN) (Pervez al., 2024) has recently emerged promising approach in scientific machine learning. Unlike traditional black-box approaches for dynamical systems (Chen et al., 2018; 2021; Kidger et al., 2021; Norcliffe al., 2020) that primarily focus on forecasting, MNN additionally learns an explicit internal ordinary differential equation (ODE) representation from the noisy observational data that enables various downstream scientific analysis such as parameter identification and causal effect estimation (Yao et al., 2024). Despite their advantages, the original formulation of MNN faces significant scalability challenges. Specifically, the computational time and memory usage severely limit the practical applicability of MNN to problems involving long time horizons high-resolution temporal data, such as climate recordings (Verma al., 2024), as the required computations become prohibitive even for the most advanced hardware.
The inefficiency stems from the matrix op- erations required to solve the linear systems associated with the MNN. In the original framework, two solvers provided: a dense solver and a sparse solver. The dense solver operates dense matrices and employs standard methods for solving linear systems, resulting in cubic time and quadratic space complexities with respect to the sequence length. This computational inefficiency makes it unsuitable for long sequences. The MNN sparse solver, on the other hand, constructs sparse matrices and uses iterative methods such as the conjugate gradient algorithm to solve the linear systems. While this reduces memory usage by exploiting sparsity, the unstructured sparsity patterns of the matrices prevent the solver from fully leveraging the GPU's parallelism potential. Additionally, iterative methods can suffer from slow convergence and numerical inaccuracies, particularly for large-scale problems.
Weekly Mean Temperature [*C]
40
20
10
Figure 4-year sea surface temperature (SST) forecasting using the Mechanistic Identifier (Yao et al., 2024) and our Scalable Mechanistic Neural Network (S-MNN).

---

Published as a conference paper at ICLR 2025
This work proposes scalable variant of MNN (S-MNN) that reduces the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear, while maintaining on-par accuracy. As result. we successfully demonstrate the real-worl well-world applicability (Figure 1) of S-MNN on long-term climate data that the original MNN failed to handle. Our main contributions are follows:
Complexity Reduction. We reformulate the original MNN's underlying linear system by eliminat- ing the slack variables and central difference constraints, and reducing the quadratie programming problem to least squares regression (Section 3.1). This results in the left-hand side square matrix having banded structure, allowing us to employ efficient algorithms (Section 3.2). The time and space complexities are reduced to linear with respect to the sequence length, making it suitable for senuene modeling.
Efficient Solver Design. We develop an efficient solver that leverages the inherent sparsity and banded structure the reformulated linear system (Section 3.2). The solver is optimized for GPU execution, fully exploiting parallelism to achieve significant speed-ups.
Long- Sequence Modeling for Science. We validate the effectiveness of our S-MNN through experiments various benchmarks, including governing equation discovery with the Lorenz system (Section 5.2), solving the Korteweg-de Vries (KdV) partial derivative equation (Section 5.3), and sea surface temperature (SST) prediction for modeling long real-world temporal sequences (Section 5.4). Our results demonstrate that S-MNN matches the precision of the original MNN while significantly reducing computational time and memory usage across the board.
2 OVERVIEW OF MECHANISTIC NEURAL NETWORKS
The Mechanistic Neural Network (MNN) by Pervez et al. (2024) consists of three components: mechanistic encoder, a specialized differentiable ordinary differential equation (ODE) solver based on constrained optimization, and a mechanistic decoder. The mechanistic encoder, realized neural network, generates a semi-symbolic representation of the underlying ODE from an input time series, effectively learning the dynamics from the data. The solver then constructs and solves linear system that equivalently describes the original system. The mechanistic decoder processes the solver solutions to the final outputs.
For tasks such as forecasting future sequences from past data, the encoder processes the input data to produce the sereector--specife semi-symbolic representation. Additionally, the encoder can be designed to overcome the limitations of the linear ODE solver by learning parameters of nonlinear basis functions (Brunton al., 201 2016a), enabling the MNN model nonlinear ODEs effectively. The decoder is optional and their necessity depends on the specific task used for training.
Formally, given multidimensional discretized time sequence 1 T2 ØT the input, the mechanistic encoder maps 1 into semi-symbolic representations of a set of linear ODEs
R
d'y x1.r) dt
Σ
c (t. xir))
of the time t dependent variable E RV with the initial conditions
d'y u, (æIrT)
(2)
dt
where V is the dimension of y, R is the highest derivative order, e, (1, .1:7) E RV and d (1, #1.T) E IR are the coefficients, and u, (œ1r)) C, (f, (I.T), d (t, x1.T, u, (wI:T), , as well as the step sizes s (x11T) for the time discretization learned by the encoder. The representation is compactly denoted {c,d, d, u, s). The MNN solver then solves the linear ODE using this representation, producing the discretized output time sequence Vl, ......: Vr. The decoder takes y1.r input and generates the final output sequence z1, Z............................... ZT.
RRV represents the initial conditions.
A loss can be computed based on Z1.7 and the target data, and the encoder and decoder networks are updated using gradient descent methods.
An illustrative example is the task of discovering a V-dimensional coefficient & RVV for a inneenddeeendet meedimensional first-order ODE in the form of
dy/dt (g (v))
2

---

Published as a conference paper at ICLR 2025
where Φ IR RV is V-dimensional non-linear basis function, g g:R. - R. is a differentiable function. this task, we first generate an initial & and specify the initial condition dyy= = I1. We fix ones where they multiplied with the first-order derivatives, and zeros otherwise. The step sizes & are determined by the dataset's time increments. During each gradient descent iteration, set (5 Φ (a:),, and then the MNN solver solves for discretized solution yı.r for Eq. 3.
We update by minimizing the loss e = Ei=t 11:: -2,11².
3 SCALABLE MECHANISTIC NEURAL NETWORKS
In this section, we define the linear system for the Scalable Mechanistic Neural Network (S-MNN) in Subsection 3.1, and present the solver's implementation and complexity analysis in Subsection 3.2.
3.1 LINEAR SYSTEM FORMULATION
A linear ordinary differential equation (ODE) system can be characterized by set of linear equations involving the successive derivatives of an unknown time-dependent function y. Formally, in a system with V variables (output dimensions), derivative orders up to R, and Q governing equations, the state discrete time points can be described by series of clauses
dr,q. E (1, , gE (I....,
where ut, is the r-th derivative of the v-th variable at f-th time point, Ct,g,,r is the corresponding coefficient for the q-th governing equation, and d is constant term.
Initial values uta.r are specified for each yt,ur up to time point Tinit (1 S Tinis S T) and derivative order Rinit (0 S Rinit I R):
Ut,ur = Ut,ur E (l....., Timit
U v
(I....,,V) v},r r Rimi..,
(5)
To ensure the smoothness of the trajectory, i.e., that the computed higher-order derivatives are consistent with the derivatives of lower-order terms, we introduce smoothness constraints. We define the forward and backward smoothness constraints using the Taylor expansions of the function time points and + respectively:
Verr Vt (L....,T 1) (L...,, (O...., R}, (6)
r)!
81) (-)))=--r Ut. (r' r)! W4 Vi
R
(I,..., V), r to. R}. (7)
p'rr
where 8; the time span between time points & and t++1
Combining the constraints Eqs. 4, 5, 6, and yields linear system. In total there are m constraints and n unknown variables where
m = TQ + Tiai (Rmit + + 2)
and T=
(8)
We arrange the unknown variables Vt,,, into a vector y E R", the left-hand side coefficients into a matrix E Rm and the right-hand side into a vector b pmm The linear system can then be compactly represented Ay b.
This system is over-determined (m > n) under typical conditions (T' > 1) and can be solved for using least squares regression. To balance the contributions of different constraints, weight the smoothness constraints (Eqs. and 7) by sT. Additionally, we introduce optional importance weights
Wgov Winit Wsmooth R. applied to the governing equations (Eq. 4), initial conditions (Eq. 5), and smoothness constraints (Eqs. and 7), respectively. This flexibility allows for emphasizing specific aspects of the model during optimization. These weights encoded into diagonal matrix W. and the solution for is then given by:
y (c, d, u, (ATWA) A Abb.
3

---

Published as a conference paper at ICLR 2025
Note that is differentiable with respect to c, d, u, and s. We provide more details the formulations of A, b, W, and y in Appendix A.1.
Our S-MNN formulation has three key differences from the original MNN formulation in Pervez et al. (2024): (1) the slack variables in the smoothness constraints are removed; (2) the forward and backward smoothness constraints
extended to the highest order (r = R), replacing the central difference constraints; (3) the quadratic programming is replaced by a least squares regression. For supplementary description of the aforementioned MNN aspects, please refer to Appendix C.
Also note that, unlike the finite difference method which approximates derivatives by discretizing differential equations, our approach formulates linear system to directly involve the derivative terms.
3.2 SOLVER DESIGN, ARCHITECTURE, AND ANALYSIS
The primary motivation for our improvement is our observation that, if we remove the slack variables, the matrix will exhibit specific sparsity pattern that be exploited for computational and memory gains. A direct implementation based on the dense matrix A using Eq. incurs cubic time complexity and quadratic space complexity due matrix multiplication and inversion. However, by analyzing the sparsity pattern of A, we find that - O (1/T) of the values in the intermediate steps are zero and do not need to be computed or stored. In more detail, Eq. can decomposed into two steps: (I) matrix-matrr multiplication M (A W) A and matrix-vector multiplication W) b; (2) solving for via the linear system M y B3. The key idea is to structure M as a banded matrix. Observing the constraints, we note that the coefficients at time point / are directly related only to those at time points t- 1, t, and + 1. By ordering the variables by we can arrange M into a banded matrix. Specifically, the variable yt.er is placed at the position ((-11) + 1) (R + 1) + r
the vector y, and the columns of matrix A are ordered accordingly. The resulting M is symmetric matrix in a block-banded form:
Mi
NT
N
M2
M
(10)
N
Ny-1
M
where each block a square matrix of size V (R+ 1). It is important to note that such a banded form is only possible after removing the slack variables from the original MNN formulation, as the slack variables introduce direct relationships between components at all time points.
For the matrix M, only the non-zero blocks Mi and N, need to be computed and stored. This can be achieved using efficient dense matrix operations with appropriately formatted dependencies c, d, u, and The matrix A does not need be explicitly constructed. We present the detailed forward pass calculations for computing M and ,3 in Appendix A.2 and omit them in this subsection. The backward pass is supported by automatic differentiation.
For solving the linear system My = 3. we propose an efficient GPU-friendLy algorithm
(Algorithm 1). Specifically, because M = A A, M positive-definite, and we can factorize M to a banded lower triangular matrix P and a block diagonal lower triangular matrix L using blocked versions of LDL and Cholesky decompositions such that
PLL P =
(11)
where P and L are partitioned into block matrices in the same form as M:
I
L
P
L
P
and ==
(12)
Pr-1 I
Lr
with each block as a square matrix of size V VR + 1) and the diagonal blocks of P as an identity matrix J. We also partition B and y into sub-vectors, B , ..... [T] and
4

---

Published as a conference paper at ICLR 2025
y yi T M/2 yr sequence of equations:
We then use forward and backward substitution to solve the following
(13) The main advantage of this algorithm is that only the blocks P. and L, in the matrices P and L need be computed and stored, which limits the computational complexities of the LDL and Cholesky decompositions and the subsequent forward and backward substitutions to be linear in T. In contrast, explicitly computing M or L- would involve dense or triangular-dense full-size matrices and should be avoided.
p L9" =3, L LT = 3", y=
Interestingly, for the backward pass of My = 3, the back prock-pppagtted gradients of M and B have elegant analytic solutions that enable further (constant factor) speed-ups compared to automatic differentiation. Assuming that the final loss is L, and given / the gradients are
ae
de
and O
M
(14)
0g
dy
a
The proof can be found in Appendix A.3. Algorithm details the backward pass. Computing 06/aß involves solving a similar linear system using the banded matrix M, and de/0M is a vector outer product that can be efficiently computed for the non-zero blocks M, and N₁. The matrices P. L, and the solution y can be stored during the forward pass and reused for the backward pass.
In Appendix A.4, we present the training (Algorithm 5) and testing (Algorithm 6) procedures for generic application of our S-MNN framework to help readers have better overview.
Algorithm 1: Solver Forward Pass
Algorithm 2: Solver Backward Pass
Input: MI.r, N1.T-1,81.7 Output: LIT, P1r-1. V1:T
Input: LITT, PIT--1 VI.T, Byle at 0t
Output:
a
1 LITT P1T-11
UMirr TN 8
DECOMPOSE (MrT, NI:T-1):
a B8..
2 yir SUBSTITUTE (LITT, PiT- BiT);;
SUBSTITUTE (LITT, (LIsT, PI.I-1: Bynr
for +- to T parallel do
D8
88 y.: 83.
DM.
end
for
to T -1 1 in parallel do
ae
28 y Vt+1
08
Algorithm 3: Decompose
DN,
03.11
Input: MI.T, No:T- Output: LTT PIT-1
end
1 LIT P1.-1 + NiT-Ni
Algorithm 4: Substitute
2 for t + 1 to T do
Input: LITT PITT-I:Q1:7
/* blockwise Cholesky */
Output: 01:r
i 1 then
for + to T do
P + P
TT
/- forward substitute */
L, L, P.
2
ar + a, - P
end
end
7
CHOLESKYY (L.); // standard Cholesky
for t to T in parallel do
ai + L, (L a,);
8 end
end
9 for
+- 1 to - 1 in parallel do /* to blockwise LDL
for
+ T 1o do /x backward substitute */
P. + P.L.
ar + a. P. a++1
end
end
Numerical Stability Considerations, An important aspect of our solver design is the numerical stability offered by the direct method of Cholesky decomposition compared to iterative methods like the conjugate gradient (CG) algorithm. Both direct and iterative methods have errors influenced by the condition number of the matrix. However, direct methods tend be more stable in practice
5

---

Published as a conference paper at ICLR 2025
because they compute an exact solution up to machine precision. In contrast, CG is iterative and can suffer from error accumulation across iterations, especially when the matrix is ill-connitioned or when the number of iterations is limited for computational reasons. This is crucial in our applications, where accurate solutions are necessary for modeling chaotic systems.
Complexity Analysis. The original MNN dense (exact) solver described in Pervez al. (2024) has time complexity O (T R2) and space complexity O (T) for time T, V variables, and I derivative orders. The original sparse (approximate) solver working via conjugate gradient has both time and space complexities O (TTV2 It²). In our proposed exact S-MNN, the time complexity is reduced by a ( factor to O (TV3 R from the MNN dense solver, and the space complexity is reduced to O (TV2R* R²) Thus, both time and memory requirements now depend linearly on the number of time points T, making the new method more scalable for longer trajectories.
4 RELATED WORK IN SCIENTIFIC MACHINE LEARNING
Scientific machine learning has emerged as transformative field that combines data-driven ap- proaches with domain-specific knowledge to model complex dynamical systems. Various specialized methodologies have been developed to tackle different aspects of this challenge, particularly in solving differential equations using neural networks and, to lesser extent, inverse problems.
Models for Prediction. Neural Ordinary Differential Equations (Neural ODEs) (Chen al., 2018: 2021; Kidger et al., 2021; Norcliffe et al., 2020) model continuous-time dynamics by parameterizing the derivative of the hidden state with a neural network. Neural ODEs, however, are constrained by the structure and sequential nature of ODE solvers and can be inefficient to train. Neural Operators (Li et al., 2020b;a; 2024; 2021; Azizzadenesheli al., 2024; Boullé & Townsend, 2023) are designed to learn mappings between infiniteddimensional function spaces, enabling the modeling of PDEs and complex spatial-tempora, patterns. However, their focus on lower frequencies in the Fourier spectrum can lead to poor prediction over longer roll-outs (Lippe et al., 2023).
Models for Discovery. The line of work on Sparse Identification of Nonlinear Dynamical Systems (SINDy) (Kaheman et al., 2020; Brunton al., 2016b;a; Kaptanoglu et al., 2021; Course & Nair, 2023; Lu et al., 2022; Rudy et al., 2017) aims to discover governing equations by identifying sparse representations within predefined library of candidate functions. However, SINDy is only a generalized linear model that does not use neural networks and can struggle with highly complex, noisy, strongly nonlinear systems. snysics-informed networks and universal differential equations (Raissi et al., 2019; Rackauckas et al., 2020) also work as discovery methods for inferring unknown terms in PDEs. Symbolic regression methods (Udrescu & Tegmark, 2020; d' Ascoli et al., 2023) constitute another line of work that aims to discover purely symbolic expressions from data.
Discussion. Mechanistic Neural Networks (MNNs) (Pervez al., 2024) have been proposed as a single framework for prediction and discovery. MNNs compute ODE representations from data which provide a strong inductive bias for scientific ML tasks. However, MNN training introduces significant challenges that require solving large linear systems during both the forward and backward passes and demands substantial computational resources. Our method addresses this scalability problem by reducing the computational complexity and enables applications on long sequences.
EXPERIMENTS
To demonstrate the effectiveness and scalability of our proposed Scalable Mechanistic Neural Network (S-MNN), we conduct experiments across multiple settings in scientific machine learning applications for dynamical systems including governing equation discovery for the Lorenz system (Section 5.2), solving the Korteweg-de Vries (KdV) partial derivative equation (PDE) (Section 5.3), and sea surface temperature (SST) prediction for modeling long real-world temporal sequences (Section 5.4). We show that S-MNN matches the precisions and convergence rates of the original MNN (Pervez et al., 2024) while significantly reducing computational time and GPU memory usage. We also compare S-MNN with other state-of-the-an methods in these experiments.

---

Published as a conference paper at ICLR 2025
5.1 STANDALONE VALIDATION
To assess the correctness of our solver in solving linear ordinary differential equations (ODEs), we conduct standalone validation. Our solver is designed solve linear ODEs directly (Section 3.1) without incorporating additional neural network layers or trainable parameters.
Experiment Settings. We select five linear ODE problems from ODEBench (d' (d'Ascoli al., 2024)-RC Circuit, Population Growth, Language Death Model, Harmonic Oscillator, and Harmonic Oscillator with Damping that are commonly used in various scientific fields such as physics and biology, along with an additional third-order ODE. Mathematical details about these ODEs are provided in Appendix B. I. For each problem, we discretize the time axis into 1,000 steps with uniform step size of 0.01 and apply our S-MNN solver.
Results and Discussion. We compare the numerical solutions obtained by our solver against the corresponding closed-form solutions. Figure 2 presents the results, where we plot the solutions y(t) along with their first and second derivatives (t) and yt(t). The numerical results from our solver closely match the analytical solutions, exhibiting negligible differences. These results confirm that our solver is capable correctly solving linear ODEs. In Appendix B.1, Table 3, we provide the exact errors for each benchmark, and the comparisons to the classic solvers RK45 (Dormand & Prince, 1980; Shampine, 1986) and LSODA (Hindmarsh, 1983; Petzold, 1983).
Circuit
Population
Language Death
Harmonic
Harmonic Damping Third-Order
vit)
10
40
0.25
W
0.1
0.0
5
0.25
-0.25
0.0
0.0
0.2
M
0.0
-2.5
5
0.0
-1
0.0
0.0
0.1
t
Closed-Form Solution
S-MNN Solution
Figure 2: Standalone S-MNN solver validation results compared with the closed-form solutions.
5.2 COMPARATIVE ANALYSIS: DISCOVERY OF GOVERNING EQUATIONS
In this experiment, we evaluate the capability of our S-MNN in discovering the coefficients of the governing equations for the Lorenz system following Section 5.1 in the origin MNN paper (Pervez et al., 2024). The Lorenz system set of nonlinear ODEs known for its chaotic behavior, making it standard benchmark for testing equation discovery methods dynamical systems. The governing equations given by
Lorenz Discovery Loss
10a
S-MNN
MNN Dense
10a
MNN Sparse
10a
(dr/dt- (y x)) = aıæ + 02Y
10-1
dy/dt = x (p- 2) = 03 + a4y + azæz
(15)
10-2
d2//t = ry B2 a6 + a7æy
200
400
600
800
1000
Optimization Step
where a E I are the coefficients to be discovered.
Dataset. The dataset generated by numerically integrating the Lorenz system equations using the standard parameters o = 10, p = 28, and / = 8/3. We use the initial condition x = over 10,000 time steps with a step size of 0.01 using the sci scipy. integgateee......mm.mmm od function from SciPy.
Figure 3: Lorenz discovery loss over first 1, 1,000 optimization steps (exponen- tial moving average factor = 0.9) using S- MNN (ours) compared with the original MNN dense and sparse solvers (Pervez al., 2024).
2 1 and integrate
7

---

Published as a conference paper at ICLR 2025
Experiment Settings, We apply our solver to the same network architecture and dataset in Pervez et al. (2024). We train the model with the default settings: sequence length of 50 and batch size 512. An instance in the batch is created by sampling a random chunk from the trajectory. Please check Appendix B.2 for more details about the batches. The training objective is to minimize the difference between the predicted and true trajectories by optimizing the coefficients d1,--.. #7. Then, assess the scalability of our method, we measure the runtime and the GPU memory consumption across different sequence lengths and batch sizes using an NVIDIA H100 GPU with 80 GB of memory.
Results and Discussion. Figure 3 illustrates the loss convergence over the optimization steps for our S-MNN solver compared the original MNN dense and sparse solvers. The S-MNN achieves similar convergence rates, confirming that the removal of slack variables does not compromise accuracy. The final discovered coefficients ag., .... d7 are presented Appendix B.2, Table 4, alongside results from the state-of-the-art method SINDy (Kaheman et al., 2020) for reference.
Table summarizes the performance and GPU memory usage. Our S-MNN solver not only maintains high accuracy but also offers substantial efficiency improvements. Specifically, compared to the MNN dense solver, our method achieves a 4.9x speedup and reduces GPU memory usage by 50% for the default setting (batch size 512, sequence length 50). The more significant improvement in runtime compared to memory usage is expected, as our approach reduces runtime from O (T*) to O (T") and memory from O (T²) to O (T", with I" denoting the sequence length. Our S-MNN solver maintains high performance even with larger batch sizes and sequence lengths where the MNN solvers run out of memory or become computationally infeasible.
Table 1: Performance and GPU memory usage comparisons between MNN (Pervez et al., 2024) and S-MNN (ours) for the Lorenz discovery experiment.
Batch Size
512
512
512
64
4096
Sequence Length
50
5
500
50
50
MNN Dense
36.4
9.5
N/A
10.0
208.1
Time per Optimization Step [ms]
MNN Sparse
104.4
76.5
589.72
80.5
406.8
S-MNN
7.4
5.5
32.2
5.5
18.3
MNN Dense
2.77
1.18
> 80.00
1.33
14.85
GPU Memory Usage [GiB]
MNN Sparse
1.69
0.93
9.832
0.96
7.96
S-MNN
1.38
1.32
1.96
1.33
1.81
Out of memory error. *Loss does not converge after a large number (200) of conjugate gradient iterations.
5.3 COMPARATIVE ANALYSIS: SOLVING PARTIAL DIFFERENTIAL. EQUATIONS (PDEs)
Next, we evaluate the capability of our S-MNN in solving partial differential equations (PDE), specifically focusing on the Korteweg-DD De Vries (KdV) equation, which a third-order nonlinear PDE that describes the propagation of waves in shallow water and is expressed as
Oy 03 3
Dy Ör
6y
0,
(16)
0t
where y (x. t) represents the wave amplitude as a function of spatial coordinate x and time Solving the KdV equation is challenging due to its nonlinearity and the involvement of higher-order spatial derivatives, making it a popular benchmark for PDEs.
Dataset. We consider the KdV dataset provided by Brandstetter et al. (2022). The dataset consists of 512 samples each for training, validation, and testing. Each sample has a spatial domain of 256 meters and temporal domain of 140 seconds, discretized into 256 spatial points and 140 temporal steps.
Settings. We model the temporal evolution at each spatial point as an independent ODE, A Re architecture (Brandstetter et al., 2022) is employed to encode the temporal and spatial dependencies in the input sequences and feed them into the mechanistic solver. The sequence length is set to 10 seconds, and the model is trained to predict the wave profile over the next 9 seconds. The model is trained for 800 epochs. We repeat the same experiment for the original MNN dense and sparse solvers as well as our S-MNN solver.
Experiment
8

---

Published as a conference paper at ICLR 2025
Results and Discussion. Following Brandstetter et al. (2022), report the testing using the rollout averaged normalized mean squared error (NMSE), defined as
1 T En (y (z, 8) 0 (x, NMSE
(17)
Σx 02 (x, ()
where y (xr is the ground truth and y (z, t) is the model output. Table 2 presents the NMSE results for MNN and S-MNN, along with ResNet and FNO results taken from Brandstetter et al. (2022). The results indicate that both S-MNN and the MNN dense solver significantly outperform the ResNet and FNO models. Our S-MNN solver achieves slightly better accuracy than the MNN dense solver. The MNN sparse solver fails converge in this experiment. We also provide visualizations for 100-second predictions in Appendix B.3, Figure 6.
Table 2: KdV prediction error (NMSE) for ResNet (Brandstetter et al., 2022), FNO (Brandstetter al., 2022). MNN (Pervez al., 2024), and S-MNN (ours). The errors are calculated on a 20-second prediction sequence unless otherwise stated.
Method
NMSE
Method
NMSE
Method
NMSE
ResNet 0.0223
FNO
0,0276
MNN Sparse
No Convergence
ResNet-LPSDDA-1
0.0200
FNO-LPSDA
0.0055
MNN Dense
0.00006
ResNet-LPSDA-2
0.0111
FNO-AR
0.0030
MNN Dense
0.00032 (40 sec)
ResNet-LPSDA-3
0.0155
FNO-AR-LPSDA
0.0010
S-MNN
0.00005
ResNet-L.SDDA-4
0.0113
S-MNN
0.00037 (40 sec)
In terms of computational performance, the training time for S-MNN is significantly reduced to 10.1 hours compared to 38.0 hours for the MNN dense solver, indicating substantial speedup. Additionally, our method consumes less GPU memory, using 2. 19 GiB versus 3.40 GiB for the original solver. The MNN sparse solver does not converge within a reasonable time frame, taking 82.4 hours and 3.07 GiB without achieving satisfactory results.
5.4 REAL-WORLD APPLICATION: LONG-TERM SEA SURFACE TEMPERATURE FORECASTING
The ability to handle longer sequences and larger batch sizes without sacrificing perfor- mance positions our S-MNN as a powerful tool for modeling complex dynamical systems. In this section, we demonstrate real-world example
Error [CC]
4
surface temperature (SST) prediction. SST exhibits long periodic features that can only be effectively captured with long sequences.
0
-2
Dataset. We use the SST-V2 dataset (Huang et al., 2021), which provides weekly mean sea surface temperatures for 1,727 weeks from De- cember 31, 1989 to January 29, 2023, over a 1 latitude x longitude global grid (180 x 360).
-4
Figure Error visualization for the S-MNN 4-year sea surface temperature (SST) prediction.
Experiment Settings. We employ S-MNN and MNN with the Mechanistic Identifier proposed by Yao et al. (2024) to predict SST data. The model leverages mechanistic layers to capture the underlying dynamics of SST. We set the default batch size to 12,960 (corresponding to 6,480 pairs grid points and their randomly selected neighboring points) and the sequence length (chunk length) to 208 weeks. The dataset is split that the latest chunk of measurements is reserved for testing while the remaining data is used for training. The model is trained for 1,000 epochs. To evaluate the scalability and stability, we benchmark the model with different sequence lengths. Besides S-MNN and MNN, we also conduct the experiment using Ada-GVAR (Locatello al., 2020), which is also used baseline in Yao et al. (2024).

---

Published as a conference paper at ICLR 2025
Results and Discussion. Figure visualizes the 4-year (208-week, prediction made by our S-MNN with the Mechanistic Identifier. Figure 4 visualizes its prediction error over the ground truth. The S-MNN effectively captures the spatial patterns of SST, demonstrating high predictive precision.
To quantitatively assess the performance and scalability, we measure the accuracy in terms of the relative MSE (mean squared error over the standardized data), well as the runtime and GPU memory usage across different sequence lengths. Figure summarizes these results.
Relative MSE (Mean Squared Error)
Time per Epoch [s]
GPU Memory Usage [GiB]
0.06
80
40
60
0.04
30
40
0.02
20
10
0.00
104 208
208
52
1727 Sequence Length [week]
Sequence Length [week]
Sequence Length [week]
S-MNN
MNN
Sparse
Ada-GVAE
Figure 5: Testing error, training runtime and GPU memory usage comparisons between S-MNN (ours), MNN dense and sparse solvers (Pervez al., 2024), and Ada-GVAE (Locatello et al., 2020) for SST forecasting, Note that the x-axis is in log scale, so both runtime and memory consumption of S-MNN increase linearly as expected.
We observe that the relative MSE averaged over both sequence length and batch size remains low for both the MNN and S-MNN solvers, with our S-MNN maintaining low MSE even for longer sequences where the MNN solvers cannot due resource limitations. A slight increase in the relative MSE for longer sequences is expected, as modeling longer-term dependencies introduces increased complexity, and the error accumulates over extended prediction horizons. Note that for the sequence length of 1,727, we use the entire dataset for training, leaving separate testing dataset for evaluation, which is why no MSE results can be provided in Figure 5. Given the log scale the x-axis, we observe that S-MNN demonstrates a linear increase in both runtime and memory consumption with respect sequence length, aligning with our theoretical complexities for time and space. In contrast, the MNN solvers exhibit much steeper increases in runtime and memory, and the memory usage quickly exceeds the 80 GiB limit our GPU for longer sequences. We are unable to run the MNN dense solver for sequence lengths beyond 104 weeks and the sparse solver beyond 416 weeks.
CONCLUSION
This paper introduces the Scalable Mechanistic Neural Network (S-MNN), addressing the scalability limitations of the original Mechanistic Neural Networks (Pervez et al., 2024) (MNN) by reducing computational complexities to linear in sequence length for both time and space. This is achieved by eliminating slack variables and central difference constraints, transitioning from quadratic pro- gramming least squares regression, and exploiting banded matrix structures within the solver. Our experiments demonstrate that S-MNN retains the precision of the original MNN while significantly enhancing computational efficiency. Given these substantial advantages, S-MNN can drop-in replace the original MNN. We believe this advancement can provide practical and efficient method for embedding mechanistic knowledge into neural network models for complex dynamical systems.
Limitations and Future Work. While our S-MNN significantly enhances scalability, certain components, such as the sequential for-loops the Cholesky decomposition (Algorithm 3) and the forward/backward substitution steps (Algorithm 4), still limit parallelism along the time dimension due to inherent data dependencies. This sequential execution can become a bottleneck when the batch and block sizes are small compared to the number of time steps, leading underutilization of GPU resources and increased CPU overhead from launching small GPU kernels. Although we have employed CUDA Graphs to reduce this overhead, the fundamental sequential nature of the algorithms remains unaddressed. For future work. we aim to develop algorithms that retain linear time and space complexities but introduce full parallelism also across the time dimension.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Throughout this work, we have strictly adhered to the ICLR Code of Ethics. All datasets utilized in our experiments are publicly available and widely recognized within the scientific community. We ensure that these datasets do not contain any personally identifiable information or sensitive content. Our work does not involve human subjects, animals, or any form of personal data collection. We have thoroughly considered potential dual-use concerns and do not foresee any harmful applications of our methods. There are no conflicts of interest to declare, and no external sponsorship influenced the outcomes of this research. All experiments were conducted with integrity and transparency.
REPRODUCIBILITY STATEMENT
We are committed to ensuring that our work is transparent and reproducible. To facilitate this, we share the source code of both our S-MNN solver and experiments part of the supplementary materials. The code documented and includes instructions for setting up the environment, running the simulations, and reproducing the results presented in paper. By making our resources openly available and providing detailed explanations, we aim to enable the research community to validate and build upon findings.
REFERENCES
Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. Nature Reviews Physics, pp. 1-9, 2024. 6
Nicolas Boullé and Alex Townsend. mathematical guide to operator learning. arXiv preprint 0rXiv:2312.14688 2023. 6
Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie point symmetry data augmentation for neural PDE solvers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Re- search, pp. 2241-2256. PMLR, 2022. URL https: ptttpp::/proceeeeee pttpp/edeeenggnsee/ee mlr. www..ppess/eeeo /v112/ v262/ brandstetter22. html.
Steven L Brunton, Joshua I. Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113((1))39322-333, 2016a. 2, 6, 20
Steven L Brunton, Joshua I. Proctor, and Nathan Kutz. Sparse identification of nonlinear dynamics with control (sindyc). Pape-PppersOnLLiine, 19(18):7007715 2016b. 6
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary
differential equations. Advances in Neural Information Processing Systems, 2018.
6
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for
ordinary differential equations. International Conference Learning Representations, 2021. 1,
Kevin Course and Prasanth B Nair. State estimation of a physical system with unknown governing
equations. Nature, 622(7982):261-267, 2023. 6
Stéphane d' A'cooi,,, Sören Becker, Alexander Mathis, Philippe Schwaller, and Niki Kilbertus. Odeformer: Symbolic regression dynamical systems with transformers. arXiv preprint arXiv:2310.05573. 2023. 6
Stéphane d' Ascoli, Sören Becker. Philippe Schwaller, Alexander Mathis, and Niki Kilbertus. Ode- former: Symbolic regression of dynamical systems with transformers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenRe- view.net, 2024. URL httpstts: /ttpss://////eeveeeeeeee www.we.wweeeeee.e.ee.eme forum? wtw.rriimmtootttte 6= GVMo. 7, 18

---

Published as a conference paper at ICLR 2025
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Compu- tational and Applied Mathematics, 6(1):19-26, 1980. ISSN 0377-0427. doi: https://doi.or science/
10.1016/071100000xx00000000---11 URL ht httpp::// htticc/eeei107777000000000000000
7, 19
A. C. Hindmarsh. ODEPACK, a systematized collection of ODE solvers, In S. Stepleman (ed.), Scientific Computing, pp. 55-64, Amsterdam, 1983. North-Holland. 7, 19
Boyin Huang, Chunying Liu, Viva Banzon, Eric Freeman, Garrett Graham, Bill Hankins, Tom Smith, and Huai-Min Zhang. Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1. Journal of Climate, 34(8):2923 - 2939, 2021. doi: 10.
1175//CL1-D-220-1666 URL https:// http::///oraaassaaaa ametsoc org.or//////...... serg/viewl http/vveeew/urass///// /ournals/ clim//3////////// 34/ /1////////IIDD// JCL/- - 0166-16666611 xm1. 9
Kadierdan Kaheman, J Nathan Kutz, and Steven L Brunton. Sindy-pi: robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A, 476
(2242):20200279, 2020. 6, 8, 19
Alan A Kaptanoglu, Jared L Callaham, Aleksandr Aravkin, Christopher J Hansen, and Steven L Brunton. Promoting global stability in data-driven models of quadratic nonlinear dynamics. Physical Review Fluids, 6(9):094401, 2021. 6
Patrick Kidger, Ricky T. Q. Chen, and Terry Lyons. "hey, that's not an ode": Faster ode adjoints via seminorms. International Conference Machine Learning, 2021. 6
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03448 2020a. 6
Zongyi Li. Nikola Kovachki, Kamyar Azizzadenesheli. Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances Neural Information Processing Systems, 33:6755-6766 -666,, 2020b.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for paramet- ric partial differential equations. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ http////ppeeeeeeeeeeeeeeeeeeeeee net forum? d==cP (wwp:--g/888/mmOO 9NQV
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informee neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 133):1227, 2024.
Phillip Lippe, Bastiaan S. Veeling, Paris Perdikaris, Richard E Turner, and Johannes Brandstetter. PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers. In Thirty Conference on Neural Information Processing Systems, 2023, URL https: cettpa://openreviee net/foounm 1 1d= 64 6811ws. 6
Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6348 6359. PMLR, 13-18 Jul 2020. URL mlr press v1 wwee11111ocaeeeee1010 html. 9, 10
https: http://pooceeeenneeeeeee
Peter Y Lu, Joan Ariño Bernad, and Marin Soljačić. Discovering sparse interpretable dynamics from partial observations. Communications Physics, 5(1):206, 2022.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems, 33: 1911-5921, 2020. 1, 6
12

---

Published as a conference paper at ICLR 2025
Adeel Pervez, Francesco Locatello, and Stratis Gavves. Mechanistic neural networks for scientific machine learning. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:/ htt.:::://pennevee...r. forum? i dww...tthhhh...h....... 2, 4, 6, 7, 8, 9, 10, 19, 20
Linda Perzold. Automatic selection of methods for solving stiff and nonstiff' systems of ordinary differential equations. SIAM Journal on Scientific and Statistical Computing. 4(1):136-144, 1983. doi: 10.1137/0904010. URL https: / http:/ooooogggooo.ooooo...ooo. 10 1137/190000000
7, 19
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, and Ali Jasim Ramadhan. Universal differential equations for scientific machine learning. CoRR, abs/2001 10000.045,, 2020. URL https: org/ abs 20010 04385.
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991. doi: https://
dbi.ogg/00101161ggp2208.8..000 10.045. URL https: www.:w/weeeeeid............... cow.teettttcooeceeeooo.ooo.cooet science/ atticle/pp1//002199991
Samuel H Rudy, Steven L Brunton, Joshua Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science advances, 1602614, 2017. 6
Lawrence F. Shampine. Some practical runge-kutta formulas. Mathematics of Computation, 46(173): 135-150, 1986. 7, 19
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16))aayy263,, 2020.
Yogesh Verma, Markus Heinonen, and Vikas Garg. Climode: Climate and weather forecasting with physics-informed neural odes. arXiv preprint arXiv:2404. 10024, 2024.
Dingling Yao, Caroline Muller, and Francesco Locatello. Marrying causal representation learning with dynamical systems for science. Advances in Neural Information Processing Systems, 37, 2024. 1,
13

---

Published as a conference paper at ICLR 2025
A THEORETICAL DERIVATIONS
A.1 DEFINITIONS OF A, b. W, AND y
A. b. W, and y are defined as follows. A, b, and W are only theoretical and they are not explicitly constructed during computation. Only y is computed.
= Let Ar.. = 1c,.q,,0. C.. ER
Let Ar.q
A
ERV(R+I),
Let A,
A And E EROVV(RH))
Let Agov
Diag (AI,..., E TQxx.
Let Ainit
[1,0] E [(Rmm+))x(+t))
Let Ainit
Diag( Ainit..... Amnit) IK Lini 1)) (B+1).
Tini times
[ 0] E IR Tint +(Rait+1)xx
Let Ainit
Let A ((++))x(++)) such that [Ai] i.j 1 -i!! otherwise.
10
ifi>j,
Let A, E (R())))((R+)) such that [Ar] [s
{0- (-si)=+// / - -()! otherwise.
10
ifi j.
Let A Diag( At
At E (RY
Let A, = Diag( A ...... A,
E (R((+))XV(RH))
V times
A -I
Let farwarh
E RTT -))(()))))
AF -1
-
AT
Let Asmonth,
Samoothh_ackwaate backward
E (V+
-I A -
Let
gow
Anit
A =
E mxn
(18)
Asmooth Amooth.backwaat
Let b,
Idd. di T E ERQ
Let boow [ by E ERQ
Let b,
[ue,v,0.... Rimel I 1l
Let b,
T b......, btv] E ERVmat+)))
14

---

Published as a conference paper at ICLR 2025
Let Omit Let
b ......
Tinal
Tina ((Rait
b b T gov D 0 07'² ER.
(19)
Let Ugoo (w2 gov.
W gov
TR.QQ
TQ times
Let Winit
[ W mit:: Wiit
IT E Tiuit (Rinit1))
Than ((Riit +1) times
[[ (Usmooth s?)²= ..... C.....,ussotti (usmooth | ERR+!
Let W
Let
w
T R (1)
times
Let Wsmooth
W
w WT-11
(TT-)))()+1),
Let w [Wgov Winit w smooth, W smooth
Let
W
diag (w) mm
(20)
Let Dr..- = (U..,,......,......
E ERR+
Let y: , "... E (((+1))
Let
91
01
E IR.
(21)
A.2 COMPUTE M AND B
Define matrix C, E (Q+(()+)) such that [Celg, = Ce.q.ur where v= == ((i--)) 1) (R ( 1)] + and r = (i-1) 1) mod (R+ + 1).
Define vector d, [dr,1, d1.,,.... d ER.
Define constant matrix U, E 1) ((++))V such that
Unles = {
if t < Tint and ((i-1) mod (R + 1)) < Riit and i = j, otherwise.
(22)
Define vector us E (VR--1) such that
Mer if S Tin and S Riuit:
[ur]; 10
(23)
otherwise,
where v - ( and r = (i - 1) mod (R+ 1).
Define constant matrix F (+++))x((--1) such that
ifi> j, Fh 1/ otherwise.
0
(24)
Define matrix S diag (s sl .!..,, (R++)
(..,,) Define matrix ST diag (-xx)) (-s:)² ..... (-si)*) E (R(++))((+1))
Define matrix S? diag (o8, 22 "..,,, 22) E ((++))
15

---

Published as a conference paper at ICLR 2025
Define matrix St E R ((++))((R+)) such that
F FSF + +
S
(ST-1)
F 1 + +S
i=
(25)
(S:)
F FSt (S,-1)
F FS + + S
otherwise.
Define matrix St - T FSt - -(S)) F S, (R(R
Define block diagonal matrix S. Diag( St si...., Si E
Then, Mr Nr, and Br can be computed as
M. w Ci Ct + whit Up w smooth St,
Vt E (I,...T)
(26)
2 N; smooth Diag( St ,.... St
Vt E (L......
(27)
V times
B. = w gov C2 de + w Un
VE [T.
(28)
A.3 GRADIENTS OF M AND 3
In this proof,
the subscripts h, J, k denote the element index, e.g., Us is the k-th component of vector M, is the element in the i-th row and j-th column of matrix M, and M,, is the row vector in the i-th row of M.
Rewrite M M-133 as
Uk Σ
[M
k.h 3r.
(29)
Differentiate ys with respect to 8,
0yk
[M-
(30)
03,
Differentiate / with respect to 3i using the chain rule,
0e 03,
de Oy: Oyk 03,
Oe
al
Σ
Σ
[M 11 k. By:
[M- -
(31)
θy
k
k
Express the derivative in vector form,
de Oy
M
(32)
0
Because M is symmetric, M is also symmetric,
dl
08 Oy
(33)
03
Differentiate ys with respect to Mij-
n M MM
-1 3m
θ [M 'Rnn
Oyk
Σ
3r
(34)
aM,
aM,,
aM,,
Differentiate of [M-1]
k. with respect to Majs
[ [M- A.N
[M-
[M-1] j.
,,,
(35)
aM.
16

---

Published as a conference paper at ICLR 2025
Substitute the derivative back into the expression,
[M- Σ Oyk
0yk 033 yj-
[M- 3, 8n
(36)
aM
h
Differentiate with respect to Mij using the chain rule,
Oe
DE Oyk
08 Oyk
0e
Σ
Yj
Σ
V/ 03,
(37)
DM
Dyk dM.,j
By 03:
Express the derivative in matrix form,
de
0e
(38)
0M
0.3
A.4 OVERALL TRAINING AND TESTING ALGORITHMS
Algorithms 5 and 6 present the training and testing procedures for generic application employing our S-MNN framework. We utilize a trainable encoder dene. a trainable decoder d training dataset Xrrain and testing dataset Nest During training. the encoder and decoder are updated using gradient descent methods. In testing, we obtain the set of decoded outputs Z and the corresponding losses L. The lines highlighted in yellow boxes are particularly pertinent to S-MNN. Specifically, during training. LrT, Pi-r-1. and ya.r computed in the forward pass and reused in the backward pass. In testing, these quantities are computed but discarded after subsequent computations, as they not needed beyond that point. The word "AutoDiff" in Algorithm 5 is the abbreviation of automatic differentiation.
Algorithm 5: Train Epoch
Input: fenc: fdecs Xtrain Output: fene; 6dec
1 for x <- a batch in Mgrain do
C, d, u, Oone (æ:);
Compute MI.7. N1.r BI.1 using c, d, s
11 Appendix A.2
Lir, PI.T-1. N1.7 SOLVERFORWARDPASS (M N1 B1r))
1/ Algoriihh 1, which calls Al Algorithms 3 and 4 Odee (y1:T); Compute loss uusing de 0g via AutoDiff;
Compute gradient
Compute gradients J0 dyur
a1 88 via AutoDiff;
88 J.. + SOLVERBACKAARDPAS (L..T, PI.T-1 VI.T Tyn
88
JM ON
// Algorithm 2, which calls Algorithm
Compute gradients 0t OE D6 at de ad du de via AutoDiff
10
Compute gradient Z0...
at via AutoDiff:
at Update denc. Pdcc using 09... 09
12
end
17

---

Published as a conference paper at ICLR 2025
Algorithm 6: Test
Input: Benc Bdec Piest 2. L Initialize 2,6 <- 2,, ;0;
Output:
2 for x <- a batch in Neest do
3
e, d, u, s- Done (z);
Compute MI.T N1.T-11 BI.T using c, d, u
11 Appendix A.2
5
Lrr, P1:1-1. V1. SOLVERFORWARDPASS (Mrr. .. 31r)
11 Algorithm 1, which calls Algorithms 3 and 4
6
# # Odee (yI:T);
Compute loss &using
8
2, + 2 U {z} (e)
9 end
B FURTHER EXPERIMENTAL DETAILS
B.I STANDALONE VALIDATION
We list the linear ODEs used for the validation experiment and their closed-form solutions. There five ODEs from ODEBench (d' Ascoli et al., 2024) and an additional third-order ODE. co. C1 C2 are constant numbers. uo y (0), ug y (0), 21 =y' (0), u2 = y" (0) are initial values. RC-circuit (charging capacitor), (co, C1, C2) = (0.7, 1.2, 2.31), (wo) (10),
y
C2 dy
co.
(39)
C1 dt
(- tt
y = cyci + (wy cpci) exp
(40)
c102
Population growth (naive), (co) = (0.23), (uo) = (4.78),
dy O, coy dt
(41)
4n exp (cot).
(42)
Language death model for two languages, (co, ()) = (0.32,0.28) (uo) = (0.14),
dy
(co + +) +
co.
(43)
dl
CD
Cp
u0
exp (- (co + t))
(44)
+ C1
co + C1
Harmonic oscillator without damping, (co) = (2.1), (no, un) (0.4, 00.03),
d2 y
0. cuy
(45)
df
un cos (lv/c) + ا sin (l//e)
(46)
Vco
Harmonic oscillator with damping, (co, c1) = (4.5, 0.43), (uo, un = 0.044).
dy
d2y
coy + Ci
+
0,
(47)
dl
dl
exp (-2.)
t
c uo + 221
( 2 4c0
-00 6))
COS
V
4cD - ef
+
sin
(48)
/ 4co
of
18

---

Published as a conference paper at ICLR 2025
Additional third-order ODE, (ug, u1 U2) (0, -1,1),
dy d²y
y'
+
0,
(49)
dt²
dt
1/ + u1 u2 + exp (-2) (ui + u2) cos
v33 - 1
v3
v3 //1
cos
+
(u1 u2) sin
(50)
In Table 3, we present the relative MSE (mean squared enror over standardized data) and CPU runtime for RK45 (Dormand & Prince, 1980; Shampine. 1986), LSODA (Hindmarsh. 1983; Petzold, 1983), and our S-MNN method. RK45 and LSODA were evaluated using the scipy. integrate. solve_lllle.. vp function from SciPy. We run all solvers on CPUs (AMD EPYC 7513 32-Core Processor). This experiment serves solver. S-MNN achieves a relative MSE below 10 in cases, indicating excellent agreement with closed-form solutions. While S-MNN may not be the optimal solver for these pure ODE-solving tasks, offers additional features, such as batched GPU processing and differentiability. which are not available in the classical solvers.
sanity check to validate the correctness of our
Table 3: Accuracy and performance comparisons between RK45 (Dormand & Prince, 1980; Shampine, 1986). LSODA (Hindmarsh, 1983; Petzold, 1983), and S-MNN (ours). The ODE problems denoted using their initial letter: RC Circuit, Population, Language Death, Harmonic, Harmonic Damping, and Third-Order. Some high-order results are applicable low-order ODEs.
Solver
R
P
H
D
T
RK45
9.3e-12
2.9e-11
2.2e-11
1.7e-10
1.2e-10
6.7e-12
y (1)
LSODA
4.5e-11
3.5e-10
9.2e-11
1.1e-09
1.5e-09
7.4e-11
S-MNN
4.8e-12 9.4e-12
2.6e-11
9.5e-08
2.1e-07
1.0e-09
RK45
1.Se-10
1.4e-10
4.4e-12
Relative MSE
s/ (1)
LSODA
7.6e-10
1.8e-09
5.9e-11
S-MNN
4.8e-12
9.4e-12
2.6e-11
7.7e-08
2.3e-07
6.8e-10
RK45
2.7e-12
(t)
LSODA
6.0e-11
S-MNN
2.2e-07
9.6e-08
7.1e-07
5.1e-07
4.0c-07
4.3e-09
RK45
1,5
1.1
1.6
5.1
6.6
3.7
Runtime (CPU) [ms]
LSODA
1.1
0.9
1.3
5.0
5.0
2.9
S-MNN
22.2
22.2
22.1
23.8
23.8
26.1
B.2 LORENZ DISCOVERY
Table 4:
Discovered coefficients for the Lorenz system using SINDy (Kaheman al., 2020), MNN (Pervez et al., 2024), and S-MNN (ours).
Method
Q1
Q2
a3
0.4
05
06
07
Ground Truth
-10
10
28
-1
-1
-8/3
1
SINDy
-10.000
10.000
27.998
-1.000
-1.000
-2.667
1,000
MNN Dense
10.00003
10.0003
27.9760
-0,9934
-0.9996
-2.6660
0.9995
MNN Sparse
-1000555
10.0061
27.7165
-0.9304
-0.9937
-2.6641
0.9990
S-MNN
-10.0003
10.0004
27.9915
-0.9968
-0.9997
-2.6664
1.0000
In the Lorenz system experiment, we generate a long trajectory starting from the initial condition as the dataset. An instance in the batch is created by sampling random chunk of this trajectory of length T. Each batch, therefore, consists of multiple such chunks, which can be thought of as sequences starting from different points along the trajectory. This approach effectively simulates having multiple sequences generated from different initial conditions, providing diversity in the training data. The default batch size of 512 ensures that the batch size is sufficiently large to
19

---

Published as a conference paper at ICLR 2025
utilize the computational resources efficiently, mitigating the limitations associated with small batch and block sizes.
Table 4 lists the discovered coefficients after final fine-tuning, along with the results from the state-of- the-art method SINDY (Brunton et al., 2016a). Our S-MNN closely matches the ground truth, SINDy, and both the original MNN solvers, with only minor differences observed.
B.3 KORTEWEG-DD DE VRIES (KDV) PREDICTION
Figure 6 illustrates a comparison between the ground truth solution, the prediction obtained from the original MNN dense solver, and that from our S-MNN. Both models produce predictions that closely align with the ground truth, demonstrating that our S-MNN effectively captures the intricate dynamics of the KdV equation. The MNN sparse solver cannot converge in this experiment and its result is not shown.
Ground Truth
MNN Dense
S-MNN
2.0
100
100
100
Prediction yix,
1.5
80
80
80
1.0
0.5
40
40
0.0
20
20
20
-0.5
0
0.10
100
100
100
80
80
80
of yix. ()
0.05
60
60
60
0.00
-0.05
20
20
20
-0.10
0
100
200
100
200
100
200
[m]
[m]
x [m]
Figure 6: Visual comparisons between the ground truth, the MNN dense solver (Pervez et al., 2024),
and our S-MNN solver for 100-second KdV predictions.
C COMPONENTS FROM THE ORIGINAL MECHANISTIC NEURAL NETWORK
The original Mechanistic Neural Network (MNN) (Pervez al., 2024) solver approximates continuous-tin dynamics through time discretization. Our modifications in S-MNN provide alterna- tive approximation methods that improve efficiency without sacrificing accuracy. While our main focus is on presenting these improvements, for completeness, we briefly describe the components from the original MNN that we have modified or discarded. The symbols and notations used below are inherited from Section 3 and they may be different from Pervez. et al. (2024).
C.I SLACK VARIABLES AND SMOOTHNESS CONSTRAINTS
Both S-MNN and MNN share the same constraints for the governing equations (Eq. 4) and initial values (Eq.5). However, unlike the smoothness constraints in S-MNN, which are formulated as equalities (Eqs. 6 and 7), the original MNN (Pervez et al., 2024) models the smoothness constraints inequalities (Eqs. 54, 55, and 56): the approximation errors bounded by slack variable E R.
20

---

Published as a conference paper at ICLR 2025
The forward and backward Taylor approximation emrors in MNN are defined as:
8 -r
Ve+l,v,
(r' (r)
(51)
R
(-8t)* (r' - -r)! 11+1,,,
pbackward P,.
yl,
Σ
(52)
For the highest (R-th) order derivative, the central difference approximation error is used instead of the forward/backward approximation errors:
E (Ut+2,0 Vt,,,)) (st St+1) M++1,rr+1
(53)
The smoothness constraints are expressed as inequalities involving the slack variable e:
forward b,urr
e; WE (I, 11, UE (L....,, rE {0, R- R-11. 1}.
(54)
Sy Ebbckwardd
c. WE fl...., 1}, UE (I.....V), re (0. ..... R R- (55)
(st + S441) R-2 geentral Se, Vt fI....., T 2}, a (L...., V}.
(56)
C.2 QUADRATIC PROGRAMMING FORMULATION
In the original MNN, the approximation problem is formulated as linear programming (LP) problem: minimize e while satisfying the linear equalities (Eqs. 4 and 5) and inequalities (Eqs. 54, 55, and 56). Specifically, the LP problem is:
minimize
subject to
A'y' 2 b',
(57)
where A' E Rm' qm+1x(++1), b' E Rm y le,y E IR++1 and m' is the total number of constraints.
However, solving ODEs using LP within neural networks poses challenges: the solutions are not continuously differentiable, and efficient solvers are lacking. To address these issues, the LP problem is relaxed to quadratic programming (QP) problem:
minimize
y y A y
subject to A'y' =b,
(58)
where A = 06,0 E R and y, E IR are hyperparameters.
T
In QP (Bq. 58), e² is minimized term. As an approximation, the absolute value operations |-| Eqs. 54, 55, and 56 are dropped. The total number of constraints m' becomes
m' TQ + Timis V (Riit + 2 1) VR (T V.
(59)
Using Lagrange multiplier Rm' the solution can be found by solving the linear system:
(m+1)x(++1)
AT
A
(60)
A'
O xmm'
X
However, the LP (Eq. 57) and QP (Eq. 58) problems are ill-defined because the number of constraints m' exceeds the number of variables n + 1 when T is large, making the problem infeasible. The square matrix in Eq. 60 is not full rank and the problem cannot be solved directly.
To circumvent this issue, the QP problem transformed into its dual form:
I Xm
A' (1+1)x(x+1)
A
b
A
3
=
(61)
21

---

Published as a conference paper at ICLR 2025
The final solution for y is then given by:
y' = (4T A) -1 (A"
b' +74).
(62)
Notably, the QP solution of y' (Eq. 62) involves inverting the square matrix M' A A' which is not banded matrix due to the slack variable e. Consequently, solving this system has a time complexity of O (T where T is the number of time steps, V is the number of variables, and R is the highest order of derivatives considered.
22


Reviews:
[{"review_id": "ggMwvjDWm9", "reviewer": "Reviewer_1WLp", "summary": "The paper presents Scalable Mechanistic Neural Network (S-MNN) designed for long temporal sequences.\nS-MNN is more efficient compared to a previously proposed solution MNN as it reduces the computational time and space to linear in the sequnce length.\nThe paper describes the algorithmic improvements to reduce the computational cost and provide experimental results", "questions": "- Are there any cases where intuitively MNN should outperform S-MNN?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem is well-motvated \n- S-MNN seems to achieve good performance while improving the computational cost \n- Detailed experimental results are presented", "weaknesses": "As discussed by the authors in the limitation section, parallelism along the time dimension can be challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FdTXQDOuk", "reviewer": "Reviewer_y7GX", "summary": "In their paper, the authors propose an alteration to the recently published Mechanistic Neural Networks (MNN), making the architecture less computationally demanding, while maintaining its predictive power. Their method, called Scalable MNNs (S-MNN), decreases computational cost primarily by removing certain slack variables present in the original method, as well as replacing a quadratic programing problem with a simple least squares regression, which has known analytical solution and can be more efficiently computed.\n\nThe authors begin by contextualizing their work among the scientific machine learning literature, then providing a brief overview of MNNs, an architecture aimed at modeling time-series data by learning an ODE in latent space, where the inputs are first encoded to represent the dynamics of the desired ODE. The solution of this ODE is then optionally passed through a decoder to map the solution to the desired output space. The authors then go on to describe their method, S-MNN, which simplifies certain aspects of the original MNN implementation. The main difference between the two methods is the linearization of the underlying ODE, which makes it solvable using a traditional least squares regression, with known analytical solution. The authors also propose a sparse method of computation for obtaining this solution, which is easily parallelizable using GPUs. Overall, their alterations reduce the time and memory complexity of the architecture so that it scales linearly with the sequence length $T$.\n\nFinally, the authors briefly describe other related work in the scientific machine learning literature, and carry out experiments to empirically validate their method. The authors employ S-MNNs on 1) a simple set of ODEs, as a sanity check for the architecture; 2) the discovery of underlying parameters of a Lorenz system; 3) solving the Korteweg-De Vries (KdV) PDE from data; and 4) forecasting sea surface temperatures from the SST-V2 dataset. Overall, the experiments indicate that S-MNNs perform comparable or slightly better than traditional MNNs, but at considerably lower computational cost.\n\nThe authors conclude by summarizing their work and outlining limitations and possible future directions of work, including parallelizing their implementation over the time dimension, which currently need to be solved sequentially.", "questions": "I list below my two main suggestions. As previously mentioned, addressing the first point is what is most likely to change my current rating of the paper.\n\n- [**Overall Relevance To Scientific Computing**] Although the authors do list related methods in scientific machine learning in section 4, it is not clear to me why improving on traditional MNNs is desirable over utilizing these other methods. Why choose to focus on MNNs? In order to answer this question, for example, the authors could run their experiments using some of the other methods listed in section 4, listing their accuracy and computational cost when compared to MNNs and S-MNNs. For example, some suggestions for comparisons are: 1) in section 5.1 and 5.2, they could run comparisons against Neural-ODEs and perhaps a traditional neural operator such as FNO or DeepONet; 2) in section 5.3 they could compare against a traditional MLP using both data and a Physics Informed loss (as detailed in line 300); 3) in section 5.4 the could compare against other autoregressive models including a traditional transformer or a neural operator. Doing so would help contextualize their work in a quantifiable way, and could help increase my score of their submission, even if S-MNNs do not beat other all other methods.\n\n- [**Use of Relative L2 Errors**] As mentioned in the previous section, reporting errors using relative L2 error instead of MSE/RMSE provides a more reliable way of comparing methods across different benchmarks, since solutions can have drastically different magnitudes. Is there a particular reason why reporting absolute MSE/RMSE makes more sense for your benchmarks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "### Originality\nThis work innovates by proposing a more computationally efficient way of implementing Mechanistic Neural Networks (MNNs), making it more scalable for longer roll-outs and numerically stable. Although the paper largely basis itself on the recent work by Pervez et all., 2024, they improve upon this method in an original way, and present practical ways of making the MNN strategy less computationally expensive.\n\n\n### Quality\nThe submission appears to be well thought-out, making precise contributions to the previous work of MNNs. The simplification of the underlying ODE solver implied by the architecture appears to yield similar performance at a computational cost 2-5 times lower, with greater grains when the sequence length $T$ is bigger. The author's computational improvements are substantial, and the theoretical arguments used to formulate and motivate S-MNNs are well crafted.\n\n\n### Clarity\nI was not originally familiar with MNNs, so there are parts of the paper that seemed unclear to me, but I think the authors overall do a decent job of introducing the important concepts and explaining how their alterations lead to computational efficiency.", "weaknesses": "### Contribution Is Specific To A Niche Method\nAlthough the ideas in the paper seem interesting, my main criticism of this submission is that their contribution is very narrow, in the sense that it improves on a method that is generally not very employed in practice. This makes their work less relevant to the scientific machine learning community. While the authors carry out several well-designed experiments comparing MNNs and S-MNNs, it is unclear to me what the advantages of S-MNNs are over the other methods described in section 4 of the paper. Potentially including some of these other methods in the experiments carried out, and comparing their accuracy/computational cost would help contextualize their work and make a stronger case for their approach. I detail some suggestions for comparisons in the \"Questions\" section below.\n\n\n### Errors Reported Using Absolute Error Instead of Relative Error (minor weakness)\nErrors should ideally be reported as relative L2 error, instead of absolute error squared. Using relative L2 error helps contextualize the performance of the method for target functions of different magnitudes. For Example, in section 5.1 the harmonic damping problem has a solution that oscillates from $\\approx -0.1$ to $\\approx. 0.1$, while the population problem oscillates from 0 to $\\approx 40$. Using relative L2 errors would also make their results more easily comparable against what is reported in other papers for comparable problems.\n\n\n### Other comments that did not affect my score:\n- [line 333] While it is great to see very close agreement between true and learned solutions, it would still be valuable to report exact errors for each benchmark, likely in the appendix.\n- [figure 4] Since all predictions look very close to the ground truth solution when eyeballing the plots, it would be great to plot absolute error as well, which could help differentiate the performance of the two methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFVZlnsHCf", "reviewer": "Reviewer_5t9U", "summary": "The authors provide an alternative method to implement ODE solving in mechanistic neural networks (MNNs) to achieve better time and space complexity without sacrificing performance.", "questions": "- I don’t fully understand what determines the sparsity level of $A$. You say 95% in typical applications, which kinds of applications yield denser $A$? IIUC, the sparsity comes from the fact that none of the Eqs 5,6,7,8 entangle $y$ that are more than a timestep apart.\n\n### Nits:\n- $V$ is undefined when introducing Eq 1, only defined later\n- maybe you can write out the definition of A, y and b once for clarity\n\n\n### To the other reviewers\nI am not an expert in this area, so take my review with a grain of salt.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- straightforward and good idea, fuelled by several computational tricks\n- strong complexity reduction!\n- clear writing\n- clear motivation\n- experiments on toys to illustrate the point and on real world data to show feasibility", "weaknesses": "- while the description of MNNs is nice and concise, i believe a reader could benefit from a small additional description of the aspects of the network that are improved upon, like the slack variables.\n- I am missing some ablations studies, for instance on the importance weights. do they do anything in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTvCeqCklr", "reviewer": "Reviewer_bGB5", "summary": "Building on mechanistic neural networks (MNN), this work proposes a strategy to efficiently solve long sequences using S-MNN. It addresses the scalability issues and reduces complexities to linear with 3 key improvements: elimination of slack variables, higher-order smoothness constraints, and least square regression. The result is a framework that can tackle long sequences with up to 4.9x speedup over MNN, and 50% GPU memory reduction, something the original MNN could not do.", "questions": "1) The paper mentions the batch size of 512 as a default setting. For the Lorentz system with initial conditions $x=y=z=1$, what makes up an instance in the \"batch\", especially since small batch and block sizes seems to be a limitation of S-MNN (line 536).\n3) The Cholesky decomposition in Eq. 12 hinges on the matrix $M$ being positive definite, which is claimed in line 201. But how is that guaranteed ? The same remark goes for Eq. 28 where the symmetric nature of $M^{-1}$ is implied but not (re)stated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Overall, the exposition is crisp, much clearer than the work it builds upon, of which the authors have given a great overview. The more direct formulation allows efficient forward pass, and a backward pass now directly supported by automatic differentiation. That said, the authors propose an analytical form that enables quicker numerical integration. The range of experiments is broad and the results are compelling, showing significant improvements over ordinary dense and sparse MNN.", "weaknesses": "1) The work does not address the trade-offs from the 3 main changes it brings, notably the consequence of dropping the slack variables. For instance, MNN notably allows **nonlinear** ODEs to be integrated with relatively ease. Is there any reason S-MNN doesn't do the same ?\n2) Limited evaluation. The original MNN showed improved performance over ResNets and FNOs, but the current method has only been compared to MNN. Particularly, in this same setting, it would be interesting to know how well models outside the MNN family perform on the KdV and long-term sea surface temperature forecasting experiments, at the very least to gauge how difficult the task is.\n3) Training and testing. During training, a number of quantities are learned. But during testing, it is not clear whether only the $y$s are solved for, and whether other matrix quantities from Algorithm 1-4 are reused. Appropriate training and testing algorithms should help clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)