Learning and Transferring Physical Models through Derivatives
Authors: Alessandro Trenta
Andrea Cossu
Davide Bacciu
Venue: Accepted by TMLR ()

Abstract: We propose Derivative Learning (DERL), a supervised approach that models physical systems by learning their partial derivatives. We also leverage DERL to build physical models incrementally, by designing a distillation protocol that effectively transfers knowledge from a pre-trained model to a student one. We provide theoretical guarantees that DERL can learn the true physical system, being consistent with the underlying physical laws, even when using empirical derivatives. DERL outperforms state-of-the-art methods in generalizing an ODE to unseen initial conditions and a parametric PDE to unseen parameters. We also design a method based on DERL to transfer physical knowledge across models by extending them to new portions of the physical domain and a new range of PDE parameters. This
introduces a new pipeline to build physical models incrementally in multiple stages.

Published in Transactions on Machine Learning Research (01/2026)
Learning and Transferring Physical Models through Derivatives
A Alessandro Trenta
Deportment of Computer Science, University of Pisa
Andrea Cossu
andrea.cossuu&dd.nniii
Department of Computer Science, University of Pisa
Davide Bacciu
dovide. Sifide.bacciu@nnine
Deportment of Computer Science, University of Pisa
Reviewed on OpenReview: https: // open:eveeepereeee.e..mmmm net/ forum? id= IbBCDDeDF7
Abstract
We propose Derivative Learning (DERL), a supervised approach that models physical systems by learning their partial derivatives. We also leverage DERL build physical models incrementally, by designing a distillation protocol that effectively transfers knowledge from a pre-trained model to student one. We provide theoretical guarantees that DERL can learn the true physical system, being consistent with the underlying physical laws, even when using empirical derivatives, DERL outperforms state-of-the-arr methods in generalizing an ODE to unseen initial conditions and a parametric PDE to unseen parameters. We also design a method based on DERL to transfer physical knowledge across models by extending them new portions of the physical domain and a new range of PDE paramcters. This introduces new pipeline to build physical models incrementally in multiple stages.
1 Introduction
Machine Learning (ML) techniques have found great success in modeling dynamical and physical systems, including Partial Differential Equations (PDEs). The growing interest around this topic is driven by the many real-world problems that would benefit from accurate prediction of dynamical systems, such as weather prediction (Pathak el al., 2022). fluid modeling (Zhang el al., 20044, quantum mechanics (Mo et al.] 2022), and molecular dynamics [Beller & Parrinello (207). These problems require grasping the essence of the system by modeling its evolution while following the underlying physical laws. Purely data-driven supervisod models often fail at this task: while being able to approximate any function, they do not usually learn to maintain consistency with the physical dynamics of the system (Greydanus et al.] 2019 Hansen et al. 2023). or fail to approximate it when only a few data points are available (Czarnecki et al. [201]]. PllysicsIInforeee Neural Networks (PINNs) (Rasssi et al.) 2019) emerged as an effective paradigm to learn the dynamics of a PDE, by explicitly including in the loss the PDE components evaluated using automatic differentiation Baydin et al., 2018). This imposos physical consistency by design and allows the use of PINNs in regimes where data is scarce. Unfortunately, PINNs suffer from optimization issues (Wang al. 2022a.
which can
lead to poor generalization WWang et al. 2021
We propose Derivative Learning (DERL), a new supervised approach to train neural networks using the partial derivatives of the objective function, as they perfectly describe its evolution in time and space (Section Like PINNs, DERL also learns the initial and boundary conditions, as they are needed to retrieve the full solution. We formally prove that learning partial derivatives sufficient to learn the evolution of the system, and we show how DERL achieves better performance than state-of the-art data-driven approaches several physical systems, by testing its generalization on unseen data points and conditions (Section

---

Published in Transactions on Machine Learning Research (01/2026)
Model 1
Model 2
Model 3
Result
TRAIN MODEL
TRAIN MODEL
TRAIN MODEL
DISTILL
DISTILL
Figure 1: How to learn and transfer knowledge from physical model. A physical model is first trained given domain. The model is not able to access the entire domain at once (the shaded area is unavailable). Later, the model is trained preserved through distillation. The same process can be applied to a time-space domain [0, x Ω. This way, DERL learns a solution incrementally in time. DERL implements both the learning and the transfer phase.
new portion of the domain, while previous knowledge is transferred and
Physical systems are usually learned in isolation. However, the physical information acquired by a model on a given problem could be used by another model trained on a related one. We believe that the transfer of information across physical models descrves more attention and is currently understudied. The advantages of transferring information across models are well-known in machine learning. Nowadays, fine-tuning a pre-trained model domain-specifie task instead of starting from scratch is the dominant paradigm.
Our approach is based on distillation (Hinton et al.) 2015)), where the physical knowledge contained in a fixed trained model (the teacher) is transferred to a student model by distilling the teacher's 's derivatives with DERL. We show that a distillation protocol based on DERL surpasses the performance of alternative protocols based on different physical information. We successfully transfer information to new portions of the domain, to new parameterization of the physical systems, and to longer time horizons (Section 5). In all cases, the resulting final model is able to solve the whole problem by only being trained on it in an incremental way. Figure T illustrates the aforementioned transfer protocol via distillation to new domains. To the best of our knowledge, this defines new approach to transfer knowledge across physical models and to study their incremental design through distillation. More broadly, our transfer protocol can be applied in multiple stages, in a continual learning fashion (Parisi et al. 2019). In continual learning, a model is trained on sequence of different tasks with the objective of learning new tasks without forgetting previous oncs. Continual lcarning has never been studied on physical systems. Our experiments show that physical models exhibit some degree of forgetting, suggesting that applying existing continual learning strategies might help mitigate this issue on long sequence of tasks. While foresee the potential impact of continual learning for case, in this paper, focus the transfer task through distillation (notably, technique widely used also in continual learning [Li & Hoiem 2016 for 2 or 3 sequential steps.
Our main contributions are: (i) DERL, a new supervised learning approach to model physical systems. Section 3 and Figure 2 provide the technical description of DERL, while Section | reports our results on a variety of ODEs and PDFs against state-of-the-art methodologies. (ii) theoretical validation of DERT. which proves that the learned solution converges to the true one when the loss goes zero (Section 3.1. Our results hold even with empirical derivatives when the analytical ones are not available. (iii) A distillation-base protocol powered by DERL, which is able to transfer physical knowledge from reference model to student one (Section 5) We discovered that distilling higher-order derivatives improves the final performance.
We hope our work can pave the way to an incremental and compositional way (Xiang et al. 2020 Soltoggio
et al.
2024 of learning physical systems.
2 Background and Related Works
Ordinary and Partial Differential Equations. Many real-world physical systems are often described through Ordinary Differential Equations (ODEs), which can be written as
(0) (1) + ai (n-)) + ... an- (()((()(( + agu(!) f(1),
(1)
2

---

Published in Transactions on Machine Learning Research (01/2026)
where (t) d'u (t) is the i-th order derivative. The order of the ODE is defined as the highest order derivative in eq. I in this case n. An n-th order ODE can always be converted into a first-order system of ODEs by considering u(t)= = [u(t), a(t),.... ................................ a (nnn1) (t)), leading to
á - ((((t) (5),
(0 = Ug.
(2)
This definition actually allows for the representation of more complex systems, such as coupled oscillators. Here, u: [0, T] + R" is a function describing the state of the system, u(0) = uo is its initial condition, and f (=((t),;;)) Rd - IR" represents the dynamics and evolution of the system based on its current state and (possibly) parameters ç. The existence and uniqueness of a solution are guaranteed under hypotheses such as Lipschitz continuity of f (Hartman 2002). Hence, eq. [1] indicates that the evolution of the physical system is completely described by its initial state and its derivatives.
PDEs describe physical systems on general domains Ω C R with boundary An through the conditions:
Flu: = 0 ((0,xr) = =2(x)
t [0, 2
(PDE)
x E Ω
(IC)
(3)
x(t,x)) = b(t, x))
x E a22
(BC)
where F is a differential operator, & E TR' is set of parameters that regulates the dynamics, and g(x) and b(t,z) are functions that define the Initial condition (IC) and Boundary condition (BC), respectively. Problems of this kind are completely determined by the PDE, which describes the evolution and propagation of information from the TC and BC to the rest of the domain through time and space (Evans) 2022).
Classical numerical methods involve diverse strategies to solve ODEs and PDES. While for the first, it might
suffice to consider the integration in time ull At) ull) (tf((((), (), PDEs often require more complex methodologies and assumptions to ensure the convergence to the true solution (Evans 2022). A common strategy involves discretizing the variables on a grid to approximate derivatives, transforming the PDE into a system of ODEs (finile differences, Ferziger& & Peric (2001). ). Similarly, finile volumes methods (Ferziger & Peric 2001) divide the space into small volumes, each one exchanging physical information and quantities with its neighbors via fluxes on their boundaries. Finally, the finite element method (Anderson et al. 2020 uses the weak formulation of PDEs, where the solution is tested against a base of functions on a mesh of points, transforming it into a complex system of ODEs. To apply classical methods, one needs to specify pre-defined grid mesh of points where the solution is calculated, which cannot be changed afterwards. Although being precisc and providing theoretical guarantoes on the convergence to the truc solution as the grid becomes finer and finer, numerical methods are limited by being tied to a specific mesh. To predict the values of the solution for other points, they need to recalculate the entire solution or to rely on approximations and interpolations. Hence, they require accurately selecting the correct mesh, balancing refinement and complexity. In fact, they can also be very slow, as these methods need to invert quite large matrices. Hence, there has been a growing interest in ML models for PDEs as they are by far the fastest at inference and allow for easy queries of any point. Their capabilities are limited only by the availability of data and, in general, fewer results on theoretical convergence.
Supervised learning methods. Recurrent neural networks (Schmidt 2019 are prototypical example of systems that evolve through time based on their previous state, which makes them a good candidate to learn solutions to ODEs. However, they only work iteratively and are prone to error accumulation or the vanishing gradient effect (Pascanu et al., 2013). Similarly, ResNets (He et al. 2016) and NODEs (Chen et al. 2019) model the residual state update as a discrete or continuous derivative, but cannot predict complete trajectories at once require an external ODE solver. Normalizing flows (Rezende & Mohamed, 2015 model dynamical systems such particles by learning their distribution with iterative invertible mappings. Neural operators (Li et al. 2020b) and their evolution (Li et al. (021) act as neural networks for entire functions by mapping initial conditions or parameters to a solution via kernel operators and Fourier transforms. While these methods are powerful and find many applications to real-world problems Pathhk et al., 2022), they require a large amount of data to generalize, are computationally intensive, and do not ensure that the solution is physically consistent. Neural Operators are also suited for a different purpose than ours, as they are not made to map point in time-space 1, æ to its value at). Our work shares similarities with Sobolev
3

---

Published in Transactions on Machine Learning Research (01/2026)
learning (Czarnecki et al. 2017 Srinivas & Fleuret 2018). which adds derivative learning terms to the supervised loss with application to reinforcement learning and machine vision. As Sobolev learning is not designed for physical systems, we adapted it by adding the 1C and BC into its loss, including a term for both u and its derivatives Du. We show that the pure derivative approach of DERL achieves a better performance.
Physics-inspired and Physics-Informed methods. PINNs (Rassi et al. 2019) incorporate PDEs that describe the underlying physical system directly into the loss. Using automatic differentiation (Baydin et al.] 2018), partial derivatives of the network are calculated on a set of points in the domain, called collocation points, and used to evaluate the PDE on the model Flá; E| to optimize it. Formally, given points [tt, ;:))s=,....NN, in the domain, the PDE residual loss is calculated by
N
[F = Na
Σ (F((():)))
(4)
3 on the network û. PINNs are used to increase the physical consistency of a model. However, PINNs suffer from optimization problems (Wang et al. 2021 and
where F(ú) is the evaluation of the PDE in equation
can fail to reach a minimum of the loss (Sun et al. 2020 To alleviate these issues, there exist met hodologls to simplify the objective (Sun et al.] 2020 or to choose the collocation points where the PDE residual is
Lau ct al. 2024). Both approaches are problem-speeific and add complexity to the optimization process. Specilic architectures for parametric PDEs are also developed (Zou & Karniadakis,
evaluated (Zhao 2021
2023 Penwarden et al. 2023) but often require multiple instances to be trained. Sobolev norms have also been
applied to PINNs for improving convergence guarantees (Son et al. 2021). Models inspired by physics exploit the formalisms of Hamiltonian (Greydanus et al. 2019) and Lagrangian (Cranmer et al. 2019) mechanics to be inherently consistent with the properties of the system, but they require the system to be conservative and to be (implicitly) formulated in the Lagrangian or Hamiltonian formalism, which is generally not the case. They also require an external solver to compute the trajectories.
Knowledge transfer. Distillation Hinton et al. 2015 was originally proposed as a method to transfer knowledge from a large teacher model to a small student model with minimal performance loss. Over time, distillation proved to be useful tool for many cases Gu al. (2023). Continual learning is one of them,
where distillation is used to avoid forgetting by transferring the knowledge from teacher trained on previous
tasks to a student trained on the current one Li & Hoiem
(2016): Carta et al. (2024). Existing reviews on knowledge transfer include (Gou et al.; 2021: Yang et al.)
2024 for an extensive treatment. We employ
distillation as a means of enabling knowledge transfer across physical models.
3 Derivative Learning (DERL)
Models that learn dynamical systems in supervised manner must be capable of simulating the evolution of the system in regions or at resolutions that are not available during training. Similarly, models for physical systems described by parametric equations are required to generalize to unseen sets of parameters. A model that is not consistent with the underlying physical laws will not provide reliable predictions under such conditions. Concretely, given a set of collocation points {=:]==1....N x, E Bd and their evaluation (u(æ:))==1....N computed according to the true function u: R - TR, there exist infinite solutions that pass through those points. We are interested in learning the one solution that is compatible with the underlying physical model. In the theory of PDEs, uniqueness results require considering functional norms and types of convergence that go beyond the (22 norm, related to the Mean Squared Error in ML. In particular, we need to consider the Sobolev space (Maz'ya ya 2011p, that is, the space of functions equipped with the norm
|ul 2 [ullwmanm) = | + Σ D'u 2 13(0))
(5)
where D'u is the I-th order differential, the set of I-th order partial derivatives. Sobolev learning (SOB, Czarnecki et al. (207)) employs this norm to better train ML. models in settings with scarce data applied to
4

---

Published in Transactions on Machine Learning Research (01/2026)
AG
PG aty
A
M
Oz
Ot
D
PIB
8u
Ou
a
an
81
dy
8t
u an
24 n
DERL
OUTL
Figure 2: Graphical representation how DERL learns function u(t,,, Partial derivatives dú dú dü 8e By
of the Neural Network (NN) output û are computed by Automatic Differentiation (AD). DERL learns the partial derivatives as independent largets, together with the initial or boundary condition wlan- We compare DERL to OUTL, which learns the function directly in the domain Ω and the boundary 00.
reinforcement learning and machine vision. In the following Sections, we show that, instead, is sufficient consider only the derivative terms in eq- [ Icading to the same thcoretical guarantees and a universality theorem, but stronger experimental results due to simpler loss. Finally, also describe new incremental paradigm to learn physical systems, employing our methodology, which alleviates PLNN optimization issues.
We now introduce DERL (fig. 2 a supervisod approach that learns from contimuous dynamical systems such as ODEs and PDEs. The popular PINN model (Raissi et al. 2019) finds the solution to a PDE by minimizing its residual [[ computed on the model, which is taken as measure of its physical consistency. DERL takes a different route. The key idea stems from the fact that the initial state and the state derivatives of dynamical system necessary and sufficient to predict its full trajectory. From mathematical point of view, two fonctions with the same continuous derivative and with point in common must coincide, in the spirit of the Fundamental Theorem of Calculus. With this in mind, DERL minimizes the following loss function:
Derivative learning
Boundary cond.
Initial cond.
I( u.) Au. [Dūū(t, Du(t, ar)l AB [[(((, b((,a=))1 A a0) g(x))
(6)
DERL's loss combines the 2 loss on the function jacobians Du, the BC and the IC, where A AB. A are hyperparameters controlling the weight of each component. The first term corresponds to the 1.2 loss the gradient Vu when u(x:) E R. The I,2 losses collocation points (Appendix AA). In the case of time-independent problems, such the Allen-Cahn equation (AC) in table [ the last term of the loss is dropped. We implement à as a multi-layer perception (MLP). DERL works even when analytical derivatives are not available, as empirical derivatives obtained with finite differences on u (Anderson et al. 2020) are sufficient to train the network. Compared to supervised approach that learns u (called OUTL the rest of this paper, fig. 2b, we will show that DERL generalizes better to unseen portions of the input domain and to unseen PDE parameterization.
computed empirically the Mcan Squared Error over a set of
Derivative distillation for transfer of physical knowledge. We leverage DERL and distillation to transfer the physical knowledge contained in a pre-trained model to a randomly initialized student or to fine-tune the teacher itself (similar to continual learning) while preserving previous knowledge. To achieve so, we compute the (frozen) teacher derivatives Dîg and the student derivatives Dūg in the domain. The student network is trained with the loss:
Letuulntt = Laystem + DåT - Düsl12,
(7)
which comprehends both the loss on current data coming from the environment (Lsystem) and the distillation loss between Leacher derivatives and the sludent derivatives. This relieves the burden of learning exclusively
from raw data and enables continual exchange of physical information across models. We will show that other distillation losses underperform with respect DERL. The MSE replaces the KL divergence commonly used in distillation for classification problems. We will show that distilling righer-orden derivatives leads substantial improvements in the physical consistency of the final model.
5

---

Published in Transactions on Machine Learning Research (01/2026)
We instantiate our incremental distillation protocol in three different configurations: i) by transferring knowledge to longer time horizons (Section 5.1 , ii) to new portions of the domain (Section 5.2p, and iii) to new parameterization of the physical system (Section 5.3). More in general, given Q the whole space considcred (domain, physical parameters, time), we partition it into non-overlapping subsets Q = U,S, We start by training a randonllyinnttializzed PINN on Q1, following the common training protocol described by Raissi et al. (019). Then, for each step > 1, we learn the new problem by minimizing a. PINN loss on the new region Q2 and we include previous knowledge through the distillation loss ||Dât - Dåslla computed on points from previous domains Uj=1....=1.... While previous works on PINNs show that they often fail to learn the true solution due to loss imbalances
Wang et al. 2022a) or implicit biases towards higher times in collocation points during training /Wang et al. 2022a) our incremental strategy can alleviate these issues.
3.1 Theoretical analysis
Neural networks are universal approximators in the Sobolev space Wmm)((2) of p-integrable functions with p-integrable derivatives up to order m (Hornik) 1991). The only requirement for the activation function to be continuously differentiable m times (as the tanh function we will use). We now prove that learning the derivatives Du, logether with the IC and BC, is sufficient to learn u. We show that minimizing the loss in equation D is equivalent to minimizing the distance between solution â and the true one Our first result involves ODEs and one-dimensional functions, which resembles the fundamental theorem of calculus:
Theorem 3.1. Let u(l) be a (continuous) function in the space (1,)((0 TT) on the interval [o, T] and a(l) neural network that approrimates it. If the network is trained such that L(u, i) - 0, then â + u.
We give the proof in Appendix B.1 The generalization of this result to higher dimensions turned out to be more challenging. We state the final result as:
Theorem 3.2. Let Ω be a bounded open set and u E (l2 a function in the Sobolev space with square norm. If the nearal network is trained such that I. u) - 0, then || u|l (1)3) -+ and || ull + 0. To have a numerical approzimation, if L(ū, u) S c, then [ (uu--u u ull S 2(C 1)c for some constant C. = C(1).. Furthermore, al the limil, the two functions coincide å= énside Ω and at the boundary.
The proof is in Appendix B.2 This result can be extended to include higher-order derivatives by adding the corresponding terms to the loss L, leading to a convergence Wmm.2 where m the highest order of derivative included. We use these theorems to prove that a neural network trained to optimize the loss in equation 6 learns the solution to PDE. We prove the following theorem in Appendix B.3
Theorem 3.3. Let u be a solution to a PDE of the form of equation E with F continuous, and let m be the order of the PDE, that is, the marimum order of derivative of u that appears in Flu). A Neural Network â trained with the loss in equation D with derivatives terms up to order m, converges to the solution of the PDE u and fulfills all three conditions.
Remark 3.4. PINNs incorporate the information on the PDE dircctly through the unsupervised PDE residual term [[F in the loss. This way, PINNs have direct access to the definition of the physical system and the underlying equation representing its dynamies. On the contrary, DERL only learns the derivatives of the true solution u through the derivative learning term in equation G without seeing the true PDE. This theorem shows that learning the derivatives is enough for the DERL to recover a solution á that fulfills the PDE, that is Flu) - 0. Hence, DERL learns to be consistent with the physical system without seeing the underlying equation. To provide an explicit example, consider a simple PDE such as Flu]- = Ut + ug = and let u be a solution related to some specified initial and boundary conditions. As the loss in equation
0 converges to zero, we have that û, + u, and ûz + u as per Theorem 3.2 Hence, Fli)- û az + ue + u Flu] = 0. Remark 3.5. While perfect consistency requires enough derivative terms up to the order of the PDE, our results show that, in practice, the loss in equation G is enough for the majority of cases, already providing better results than the other baselines. This clearly shows that learning just u, as in most supervised approaches, while derivatives already provide sufficient and richer information on the system.
When analytical derivatives are not available, we use empirical derivatives obtained via finite differences du
(a:) ufæ+hee u((e) where e is the unit vector in the direction of x; and h is Anderson al. 2020 dz,
6

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Summary of the tasks we consider in the experiments.
Experiment
Equation
Description
Tasks
Generalize in-domain
Ceneralize to new parameters g
Allen-Cahn
(AC) Mfeer Myu] uuu² 1) f(g)
Socond-order PDE
Transfer to larger donsin
Transfer to new parameters E
Continuity
(CO) du + V (er)
Time-dependent PDE
Generalize in-domain
(NS.M) [Du] u Du Vp
Navier-Stokes
System of PDEs (Appendix)
Generalize in-domain
(NS.I)V u
Pendulum
(DP) + + sin(n) L
ODE
Generalize to new initial conditions (Mo, Miy))
Transfer to larger timespan
Korteweg-de Vries
(KdV) a wile + Pigae 0
Third-order PDE
Higher-order distillation
small positive real number. These approximations introduce errors, but the following result ensures that with small enough h, these derivatives are very similar to the true ones. The proof is in Appendix B.4
Theorem 3.6. Let u E (R") and let D2, u be the difference quolient
D2,uaa = u(x + he,) u(x)
(8)
h
where e₁ is the unit vector in the x; direction. Then, we have that [|D ullEa(R>) S llan, ll (R())) and D u -> Uxx
in TR*),, which means that empirical derivatives converge to weak (or true) ones as 6 - 0. The same results hold for any open set Q2 with the convergence being true on every compact subset of Q. In practical terms, empirical derivalives converge a.e. for every point distant at least h from the boundary.
The approximation used to calculate the derivative is of first order O(h). If sufficient data is available, one can further improve precision by considering higher-order approximations, such the central scheme (Anderson et al. 2020), which has an O(h?) error, beyond.
Remark 3.7.
4 Learning Physics from Data
We validate DERL on a set of dynamical systems and physical PDE investigating the generalization capabilities novel set of domain points (section [41] and novel set of parameters for parametric equations (section 4.2 During the training phase, the models did not have access to the set of points and parameters used to evaluate their generalization. Table [ summarizes the systems we consider, with additional results in appendix C For each experiment, we compare 4 approaches applied on a Multi-Layer Perceptron (MLP, Goodfellow et al. (2016)). Each of them learns the IC and the BC, together with a loss term for the domain 12 which is specific to the approach: (a) DERL (ours) optimizes equation 0 (b) Output Learning (OUTL) minimizes |/u - ما; (c) our extension of Sobolev learning (SOB) (Czarnecki el. al., 2017 to dynamical systems include the loss term for both and Du. For comparison, we also consider (d) OUTL+PINN, which consists of OUTL with additional PINN loss term the equation to improve its performance in the domain. The full description of the basclines and losses is in Appendix A We evaluate all approaches with two metrics: (i) we measure the accuracy of the prediction by ||u - Al|2 the I,² distance between the true function u and the estimate à; (ii) considering MLPs as functions from the to the domain of u, we mcasure how well a model satisfies the underlying physical equation Flu] 0. To this end, we compute the 2 norm of the PDE residual of the network [[ 2|, as typically done with PINNs.
4.1 Generalization to new domain points
We test the ability of all approaches to learn the solution to a PDE in a domain Ω. A fixed set of collocation points used train the models, which are then tested on an unsoen grid of points at higher resolution.
The code to reproduce the experiments available at
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Results for the Allen- Cahn and continuity equation.
Model
Task
||u - a||a
|||a]]| x x10
Task
||u - alIp
|| 12
x10
x10
x10
DERL (ours)
Allen-Cahn
6.836
1.930
Contimuity
9.059
2.247
OUTL
56.95
11.41
8.749
3.762
OUTL+PPNN
39.13
7.105
9.378
2.8975
SOB
9.448
2.469
16.52
2.881
Allen-Cahn Equation. The Allen-Cahn equation (AC in table 1 is a non-linear PDE in the domain [-1,1]3. We consider x = 0.01 and the analytical solution u. = (i:(())))))))))))))))) sin(Tyy. The BC, the external force f, and the partial derivatives are calculated from u using the PDE. The MLPs, which prodict u(x, y) from (r, have 4 layers with units (similar to Raissi et al. (2019)) and tanh activalion and are trained for 100 epochs with the BFGS optimizer (Li & Fukushima 2001 on 1000 random collocation points. Additional details are provided in Appendix C.1
Table 2 shows the relevant. metrics for this task. DERL out purpeross all the other approaches in learning the true solution surpassing OUTL by a factor of 10. Furthermore, DERL achieves a lower PDE residual than PUNN. This shows that lcarning the direct solution u is not as effective as DERL, not even when leveraging PINNs as an additional source of information. DERL also outperforms SOB, showing that derivatives contain all the needed information, while adding other terms to the loss can interfere and deteriorate the performance. This is also an empirical confirmation of our theoretical results. We finally report that DERL is the approach that best approximates the true derivatives (l)Du Dū surpassing OUTI and OUTL++PINN by a factor of 5 and improving on SOB. This is, of course, expected as DERL's loss target includes the derivatives. We provide additional results on this task in Appendix C.1
Continuity Equation. We consider the continuity equation (equation CO in table D time-dependent PDE describing mass conservation of matter in velocity field. The domain of the unknown density u(t, y) is the 2D plane region [-1.5, 1.5]² for & E [0, 10). The velocity field v is given by va y) (- (-,, The density is null at the boundary and the IC is given by 4 Gaussian densities (see Appendix C.2 as in Torres et al. (2024). The reference solution is calculated using the finite volumes method (Ferziger & Peric 20011) (.01).The The MLPs, which predict u(t, x2 from (t, x. y). have the same architecture as the Allen-Cahn experiment and are trained for 200 epochs with batch size 128 and Adam optimizer (Kingma 2014p. Partial derivatives are empirical and were calculated by finite difference approximation. See C.2 for further details and results.
Table 2 reports the 2 error the solution u and the PDE residual L norm calculated on the finest grid available. DERL is the most effective method: the distance from the ground truth is second only to OUTL (although still comparable) and almost two times smaller than SOB. We conjecture this is due to its more complex optimization objective with possibly conflicting loss terms for u and Du. Plus, DERL is clearly best in terms of PDE residual, surpassing even OUTL+PINN, which means our model is more consistent with the underlying equation on domain points we have not trained the model on. On a side note, we report that the reported in Appendix C.2 These
PINN loss alone fails to propagale the solution through time correctly, results for PINNs are aligned with Wang et al. (2022a), where the authors conjectured that PINN's gradients are biased towards high values of f with the model failing to propagate information correctly from the IC.
Navier-Stokes. We now consider a well-known system of PDEs: 2D time-independent Navier-Stokes equations made of the momentum (NS.M) and incompressibility (NS.I) equations in Table [ The domain is (x, y) E [-1, 1] x [-0.5, 1.5) with the viscosity coefficient set to 50 The particular solution we work with is the so-called Kovasznay flow (Drazin & Riley 2009), representing the flow around two-dimensional grid, with the analytical form given in appendix C.3] Training data is sampled from a regular grid of points with dx = dy = 0.01, while testing data comes from a finer one, with dr dy = 0.005. We work with MLPs with layers of 64 units and tanh activation, trained with the LBFGS optimizer for 100 epochs. Here, we also measure the error on vorticity w V x u sin z sin ye 2vt2 which is a vector that describes the rotating motion of fluid at given point and is of particular interest for turbulent flows such this experiment.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Results for the Navier-Stokes Kovasznay flow experiment.
Model
((up)) - (a, 0)1
||w @||2 x10
(NS.M) 1F[0,
(NS.I) ||[
x10
x10
x10
DERL (ours)
0.6719
0.6611
0.8810
0.5945
OUTL
9.201
106.6
10.58
10.30
OUTL++PINN
3.331
5.042
0.8465
0.5729
SOB
1.125
1.077
1.660
1.177
15
OUTL DERL
155
OUTL+PIN -DERINNN-EELL -DELL
15-
506 DEAL
0.014
0.000
0.005
0.005
0.004
0.002
-10
-0.5
0.0
1.0
-0.3
0.0
1.0
-0.5
-1.0
-0.5
0.0
1.0
Figure 3: L error difference in the learned solution (ū, between each methodology and DERL. The blue area is where DERL performs better than the comparison.
Numerical results are available in Table 3 DERL is the best overall model, achieving the lowest L errors on both the solution u, p and the vorticity w. DERL also follows closely the PDE residuals of OUTL+PINN, while OUTL alone fails to reproduce the vorticity of the solution. Figure [1 shows the difference between the method's error and the one by DERL, with the blue color meaning DERL is performing better in the region. This figure clearly shows that learning only u does not provide enough information on the physical systems, especially in this turbulent case.
4.2 Generalization to new ODE/PDE parameters
We now shift towards more complex tasks involving solutions to parametric ODEs and PDEs of the form F(u; E) = 0, where ş E R the set of parameters. The objective of the models is to generalize to a set of parameters unseen during training. A supervised machine learning pipeline is performed where the set of parameters (and their solutions) is divided into training, validation, and testing. We tuned the models using the training and validation sets, with the final results given by the testing set. All models take input the spatiotemporal coordinates and the parameters (1, E).
Damped Pendulum. We consider the dynamical system of a damped pendulum with state equation (DP) in table | Additional details on the experimental setup are available in Section C.4 We sampled total of 50 trajectories from different starting conditions (wo, úo), which act as parameters: 30 are reserved for training, 10 for validation, and 10 for testing. Each trajectory was sampled every At = 0.01s. The MLPs, which predict u(I; uo, úo) from (1, un, úp), have MLP with 50 units each and tanh activation, and are trained for 200 epochs with batch size 64. Initial states are sampled randomly in (uo, úo) [- 3] x [-1.5, 1.5). In addition to the previously-used metrics and the angular velocities || ||u- [lu-āll2, we also monitor:
||[[E]]|2 (formally defined in Appendix C.4.Ip, the PDE residual which describes continuity and differentia- bility of trajectorics with respect the initial conditions. High values of [|][ū]]l2 imply predicted trajectories do not change smoothly when varying (ug, uo), which is not consistent with the real system.
The error in the learned field at t = O, that is, the distance between true and predicted derivatives |[F[]]ll for every possible initial state in the domain. This measures how well the models predict the dynamics for different initial conditions.
We report numerical results in Table ! Figure La shows a direct comparison of the errors in the learned field Flūo] at t in the whole domain. We computed the local error on grid points for each method. Then, for each method, we computed the difference between its errors and DERL's error, such that positive values (blue) are where DERL performs better. DERL is second best and very close to OUTT at predicting the
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table Numerical results for the damped pendulum.
Model
||u - a
|| 81
|[[ x10
[|]]]] x10
|F
x10
x10
x10
DERL (ours)
1.069
1.296
1.277
9.244
4.445
OUTL
0.9607
5.766
5.452
20.55
8.233
OUTL+PINN
1.118
7.614
7.070
30.07
10.65
SOB
1.133
2.824
2.803
15.17
8.054
OUTL+
50B- DERL
HNN DERL
LNN DERL
0.8711
0.3509
0.7630
0.6550
-8 -01178
0.3469
- 0.0889
0.4388
0.3308
0
- 00011
222
1146
0.2710
-1
0 06360
0.1015
0.4510
-1
0
1
11
0
0
-1
1
(a)
(b)
Figure 4: Pendulum experiment. 2 error difference in the learned field at t between each methodology and DERL. The blue area is where DERL performs better than the comparison.
trajectories and outperforms by large margin all other approaches in predicting the angular velocity of the pendulum. The trajectories predicted by DERL have also the lowest F[o] and g[u] norms. This shows that among all models, DERL's predictions are the most physical ones and the most consistent with the underlying equations. Figure daj shows that DERL learns more accurately the field for every initial condition. For this task, we adapted Lagrangian Neural Networks (LNN) [Cranmer al. 2019) and Hamiltonian Neural Networks (HNN) |Greydanus et al. 2019 to the damped pendulum case. We train them on the conservative part of the field (see C.4.2 for details), where they excel as they are specifically designed for conservative fields. Unlike DERL, they also require trajectories to be calculated through external solvers. Remarkably, DERL outperforms both LNN and HNN (figure 4b) on the learned field: DERL scores an error of 0.4445, against 0.4470 and 0.4496 for HNN and LNN, respectively.
pargmee Allen-Cahn Equation. Inspired by Penwarden et al. (002), we consider a paramet experiment on the Allen-Cate equation (AC). The parametric component the external forcing f, calculated from the analytical solution u(x, g) = E(r-000.7 (in((=))))ss(xa)))) (inn:==)sin(T)))) with parameter & R. We reserved 8 values of for training, 2 for validation, and 5 for testing, and we used 1000 collocation points in the domain. The architecture is the same as in the Allen-Cahn experiment in Section 4.1 For further details and the complete setup, see Appendix C.5 Numerical results on the testing parameters are reported in Table 5 while further results are available in Appendix C.5.1 DERL always scores the best results, highlighting both metrics,
strong generalization abilities to unscen parameterization of the PDE. OUTL underperforms showcasing that learning from the raw data often leads to misleading solutions that do not follow the underlying physical laws. SOB underperforms as it again struggles to optimize loss with both and Du.
Double- parameterr Allen-Cahn equation. We now increase complexity by considering dual-parameter version of the Allen-Cahn equation (Zou & Karniadakis 2023). The analytical solution is given by (( ΣE sin(jxx))sin(jmz) from which the external forcing is calculated. We selected 15 random couples of the parameters (2) E [o, 1]2 for training, 5 for validation and for test. Numerical results for the test parameters are reported in Table 5 Once again, DERL is consistently the best model for every couple of parameters (see Appendix (.5.2), performing at least two times better in predicting u for unseen ones. Learning u through OUTL leads to very poor performance terms of physical consistency, even with the additional help of the PINN loss.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Results for the parametric Allen-Call experiments, averaged over test parameters.
Single- Parameter
Double-parameter
Model
||u
0|/2
|||
||u
0//2 x10 3
|||
x10 8
x10 -2
x10 2
DERL (ours)
5.441
1.730
9.628
1.467
OUTL
8.486
11.67
19.13
8.050
OUTL+PINN
6.413
2.466
27.70
8.446
SOB
9.736
2.276
21.26
1.477
5 Transferring physical knowledge
We have shown that DERL. is capable of extracting the relevant physical features from data. now show how we can take advantage of this to transfer physical information between models through distillation (Hinton ict al. 2015), oventually building physical models incrementally. We set up our transfer experiments by dividing the whole training experience into 2 or incremental steps. Each step extends the physical problem to i) new portions of the time domain, ii) to new portions of the space domain, and iii) to new parameterizations of the PDE. For these incremental training experiments, assume unsupervised setting where ground truth data is not available, and the models have to find a solution only by the definition of the PDE problem, as PINNs do.
The physical models that we train and use as teachers are all PINNs. For the distillation component in equation y we tested the DERL, SOB, and OUTL losses. During the first step, we train a model from scratch on the given problem. From the second step on, we consider the teacher the model trained on the previous step, and we transfer its knowledge to a student model. The student is either a randomly initialized new model (from-scralch in (ables) or the previously trained model (conlinual), being fine-(uned We use our distillation approach to preserve the teacher's knowledge while the student learns the current step.
As a comparison, we train three additional PINN models. PINN-noddistillttonn updates the teacher with the PINN loss on the new domain, but without distillation on the previous domain, and is expected to forget previous information. PINN-full is a PINN trained on the full domain Ω without the iterative distillation process. This model is trained for the total sum of epochs of the other methodologies for a fair comparison. In continual lcarning, this joint training method is often taken as an upper bound for the expected performance of the final continual learning model Masana et al. (2023). Finally, we consider PINN-replay, which updates the teacher with the PINN loss on the new domain and a subset of points (20%) from the previons domain. This is our closest resemblance to an Experience Replay setting.
5.1 Transfer to wider time horizons
In this first experiment with transferring, we aim to extend the working domain of a PINN in time. We consider the 1D Korteweg-de Vries equation (KdV) in Table [ The spatial domain is the interval [-1, I]. while the entire time domain is [0, 1]. We first train a PINN in the time interval [0, 0.35] and we collect its evaluations û, Då on the training collocation points in a dataset. Then, a second model is trained with the combination of the PINN loss on the second time interval [0.35, 0.7], and a distillation loss on the old collocation points. This procedure aims transfer old knowledge this new model, while discovering the solution in new interval of the time domain. Hence, this model should be able to learn the solution in the time-space domain (t) E [0, 0.7] x [-1, 1]. Finally,
repeat this procedure one more time, extending the model to the time interval [0.7, 1]. As we will see, the final model has correctly learned the solution in the entire time-space domain of [o, 1] x [-1,1]. For this task, each model is an MLP with 9 layers of 50 units and tanh activation, due to the more complex nature of the KdV task. Training phases last 30.000, 30.000, and 40.000 steps each with the Adam optimizer and exponentially decaying Ir, starting from 0.001 with factor 0.9. The PINN-full, PiNN-no-ditiilationn, and PeNlar models train for the sum of these steps, that is 100.000, with the same optimizer and learning rate schedule.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Results for the physical knowledge transfer experiments. L error and PDE residual on the whole domain (or on test parameters) the final stage of incremental training.
Larger timespan 5.1
Larger domain
New parameters
5.3
Model
||u ú/|2
[|
||u - a||p
||||
||u
a||a
[|
x10
x10
x10
x10 3
x10
x10
from-scratch
3.331
2.932
3.332
7.453
7.736
7.537
DERL
continual
2.379
1.731
2.377
3.873
11.76
7.275
from-scratch
9.302
8.966
6.850
15.10
9.757
10.27
OUTL
continual
9.334
7.571
3.399
8.995
11.49
11.87
From-scratch
3.662
4.327
5.151
9.480
11.03
11.63
SOB
continual
2.535
1.536
2.642
5.284
8.840
7.645
full PINN no distillation
PINN
9.801
1.730
2.356
6,275
14.96
6.302
15.19
18.29
247.3
245.6
31.87
16.14
PINN
replay
5.371
1.889
5.281
10.54
13.72
6.289
Prediction Error
SOB continual
10-1
OUTL continual
PINN full
PINN dist.
replay
Loss
10-2
20000
40000
60000
80000
100000
Training Step
Figure 5: Prediction crrors for the KdV transfer experiment for the continnal models and the PINN variants. Vertical lines represent when a new step of incremental training is initiated.
Numerical results are available in Table DERL outperforms all other methodologies in learning the solution to the PDE, either starting from-scratch or by fine-tuning the teacher model. Most importantly, our incremental methodology, coupled with the DERL distillation loss, improves the results by three times compared to standard PINN. Figure shows the prediction error in the whole domain as function of the cumulative training steps for the continual models. A similar plot including the from-scratch models is available in Appendix D.3 These losses follow the typical behavior of contimual learning, where the loss increases at the beginning of each phase and subsequently decreases, with the difference that performance is measured the entire final task. The results clearly show that the PINN reaches minimum the beginning of training and cannot improve, while our incremental approach performs almost order of magnitude better in the end. While PINN-replay improves on the bascline, using DERL to distill the teacher derivatives performs better to retain knowledge compared to experience replay. We notice that, even for transferring, it is not enough to distill the model's outputs, both in terms of consistency and prediction error. Derivatives are both necessary and sufficient to distill the physical knowledge correctly. Finally, we see that incrementally training a PINN without distilling previous knowledge leads forgetting of the learned solution. See Appendix D.3 for additional figures.
5.2 Transfer to new domains
We consider the We leverage our transfer protocol extend the working domain of model (Figure 1
Allen-Calh equation (AC) from Section 4.1 We first train PINN in the lower-left. quadrant of the domain
12

---

Published in Transactions on Machine Learning Research (01/2026)
([-1.0) x ()), We evaluate û, Dū on the training collocation points. Then, a second model is trained with a combination of the PINN loss on the lower-right quadrant ((0, x [-1,0], 0)), and a distillation loss for the old points. The resulting model should be able to learn the solution in the lower half of the domain ([-1.0) x O)) The same procodure is repeated to extend the working domain of the model to the upper half ([-1,1] x [0, I)). The final model should be a good predictor the entire domain ([-1, x [-1,1]) We used the same architecture as Section 4.1 and 100 epochs per step with the BFGS optimizer (Appendix D.4].
The numerical results in table 0 show that DERL enables the best-performing distillation, achieving the lowest error and the best PDE residual in both the from-scratch and continual case. DERL even beats the PINN trained on the whole domain for the same number of epochs, especially in terms of PDE residual. This is surprising, as the PINN trained on the whole domain would tipically be used as an upper bound in continual learning (the so-called "multitask" model) (Masana et al. 2023 Instead, DERL surpasses its performance, showcasing promising continual learning abilities.
5.3 Transfer to new PDE parameters
We consider the Allen-Cahn equation (AC) with two parameters (2), as in Section 4.2 We first trained a PINN on 1000 random collocation points for 10 couples of E2). Then, on the remaining 10 couples we trained a new model, either from scratch or by fine-tuning the previous one. We train with a PINN loss on the new parameters and we use distillation with DERL, OUTL or SOB.
Table 6 reports the L errors and PDE residuals on the test parameters for both the Prom-scrateh and continual cases, DERL proves to be the best distillation protocol. It also surpasses all PINN models, except for the full and replay PINNs the PDE residual metric, whilst being the closest to match its performance. This again shows our method's potential to transfer physical information across different instances of PDEs.
5.4 Distillation of higher-order derivatives
Finally, we investigate the effects of distilling higher-order derivatives. We consider the Korteweg-de Vries (KdV) equation (KdV in table a third-order non-lincar PDE. A pre-trained PINN teacher distills knowlodge to students with icantical architecture. We tested Hessian learning (HESL), which distills second-order derivatives and its combination with DERL (DERL+HESL) and SOB NOB+HEEL), The full setup and results can be found in section D.3 Here, we summarize the main findings. We observe that OUTL fails to distill effectively, as its PDE residual of 17.366 is two orders of magnitude larger than the other models. Interestingly, the pest-performing methods, HESL and DERL+HEST, did not access u directly. They achieve the lowest PDE residual (0.1915 and 0.1931, respectively) which is comparable to the PINN PDE residual of 0.1664. Both approaches score the same error on u as the teacher model, and even a lower BC loss.
6 Conclusion
We proposed DERL, a supervised methodology to learn physical systems from their partial derivatives. We showed theoretically and experimentally that our method successfully learns the solution to problem and remains consistent with its physical constraints, outperforming other supervised learning methods. We also found DERT to be effective in transferring physical knowledge across models through distillation. We leveraged this capability to extend the working domain of a PINN and to generalize better across different PDE parameterizations. Finally, we showed the advantage of distilling higher-order derivatives. Our work shows promising intersections with continual learning, as DERL's 's distillation allowed us to build physical models incrementally, showing minimal to no forgetting of previous knowledge. In the future, learning physical and dynamical systems may not always require starting from scratch. Rather, the learning process may be based on the contimual composition and integration of physical information across different expert models. A more general and flexible paradigm that could lead to data-efficient and sustainable deep learning solutions.
Acknowledgments
This work has been supported by EU-FIC EMERGE (Grant No. 101070918).
13

---

Published in Transactions on Machine Learning Research (01/2026)
References
Dale Anderson, John C Taanehill, Richard H Pletcher, Ramakanth Munipalli, and Vijaya Shankar. Compu- tational fluid mechanics and heat transfer. CRC Press, Boca Raton, FL, 4 edition, December 2020.
Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey, Journal of Machine Learning Research, 18(153):1 2018.
URL http:///mw.rr..rrpppreers/s///// wttp://apprr/////114446881
Jörg Behler and Michele Parrinello. Generalized neural-network representation of high-dimensional potential- energy surfaces. Phys. Rev. Lett., 98:146401, Apr 2007. doi: 1e1100/PhyseeLLet...... 9...............666066.00 URL
https://link: aps. org//oi/1/..oooooooo htt.:///sayyaeeeettt.eoo. 146401
James Bergstra, Daniel Yamins, and David Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Sanjoy Dasgupla and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 115-123, Atlanta, Gcorgia, USA, 17-19 Jun 2013. PMLR. URL mlr wttpl/prreee//////rrtrrree html
httts::/pprceeedee
Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, Davide Bacciu, and Joost van de Weijer. Projected Latent Distillation for Data-Agnostic Consolidation in distributed continual learning. Neurocomputing, 598: 127935, September 2024. ISSN 0925-2312. doi: 10. 10.1016/j n0.1006/j..eucom.222 wtt.uoom.202.11299900
Ricky Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 2019. URL btttpp://arxxv.ogg/abb//////
07366
Miles Cranmer, Sam Greydamus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural net works. In ICLR 2020 Workshop on Inlegralion of Deep Newral Models and Differential Equalions, 2019. URL https://opearevie..net http://wwwweeaeeeeooooooood/aaaa
Wojeiech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascamu. Sobolev training for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https::/prooeedding mww.dddnnnnn..ii.....oo.m cetrrippsccpppperrreeeeee filess/aper//01//881
P G Drazin and N Riley. London mathematical society lecture note series: The Navier-stokes equations: A classification of flows and ezact solutions series nember 334. Cambridge University Press, Cambridge, England, November 2009.
Lawrence Evans. Purtial differential equations. American Mathematical Society, Providence, RI, March 2022.
Joel H Ferziger and Milovan Peric. Computational methods for fluid dynamics. Springer, Berlin, Germany, 3 edition, November 2001.
Herbert Goldstein. Classical mechanics. Pearson Education, Philadelphia, PA, 2011.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning- MIT Press, 2016. http://wwr deeplearmingbook.co org.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge Distillation: A Survey. International Journal of Compuler Vision, (29(6))1789 1789-199991 June 2021. ISSN 1573-1405. doi: 10.1007/ sl s11263-021-014553-z
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In H. Wallach, H. Larochelle, A. Beygelzmer, d' d'Alcé--Bue, E. Fox, and Garnett. (eds.), Advances in Neural
Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL |ttps//r/ppr
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yuxian Gu, Li Dong, Furu Wei, and Minlic Huang. MiniLLM: Knowledge Distillation of Large Language Models. In The Twelfth International Conference on Learning Represenlations, October 2023.
Derek Hansen, Danielle C. Maddix, Shima Alizadeh, Gaurav Gupta, and Michael W. Mahoney. Learning physical models that can respect conservation laws. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (ods.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12469 12510. PMLR, 23-29 Jul 2023. URL
mlr. http://.......r.s//eeeee wwwps/veeee////
Philip Hartman. Ordinary Differential Equations. Society for Industrial and Applied Mathematics, January 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recogrétion (CVPR), pp. 770-778, 2016. doi: D1.1199/VVR.2016..0
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXio, 7200344
00s/1553.02531,, 2015. URL
Kurt Hornik. Approximation capabilities of multilayer feedffrwwrr networks. Newral Networks, 4(2): URL https:
1991. ISSN 0893-6080. doi:
//www www..ccieneeirrect cow ottp:///cacccrrrrttre//////////0
Ameya D. Jagtap, Elhan Kharazmi, and George Em Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems.
Compuler Melhods in Applied Mechunics and Engineering, 365: 13028, 2020. ISSN 0045-7825 doi:
https://doi.org/ 10. w0t.://.c....///2....... 113028. URL p11/80045782520322227
Dicderík P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
Gregory Kang Ruey Lau, Apivich Hemachandra, See-Kiong Ng, and Bryan Kian Hsiang Low. PINNACLE: PINN adaptive collocation and experimental points selection. In The Twelfth International Conference on Learning Representations, 2024. URL http:////pppareeeeseoooooorrrtae
Dong-Hui Li and Masao Fukushima. A modified BFGS method and its global convergence in nonconvex minimization. J. Comput. Appl. Math., 129(1-2):15-88) 35, April 2001.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning, 2020a. URL. https://arxiv
Zhizhong Li and Derek Hoiem. Learning Without Forgetting. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, Lecture Notes in Computer Science, pp. 614-629, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0. doi: 10.1007/998-333114466499
37.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, 2020b. URL https://arxxv..org//a///00. 03485
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL httpss://pearrvier..cm wtt.errorrre c8P9NQVtmnO
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and Joost van de Weijer. Classenncremental Learning: Survey and Performance Evaluation on Image Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-55 -5533,,,,,333 May 2023. ISSN 1939-3539. doi: 2tA.T/APM.2.222.223333
15

---

Published in Transactions on Machine Learning Research (01/2026)
Vladimir Maz'ya. Sobolev spaces. Grundlehren der mathematischen Wissenschaften. Springer, Berlin, Germany, 2 edition, February 2011.
Yifan Mo, Liming Ling, and Delu Zeng. Data-driven vector soliton solutions of coupled nonlinear schrödinger equation using a doep learning algorithm. Phys. Lett. A, 421 (12773))))1277739 January 2022.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tunanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael Jordan, and Ion Stoica. Ray: a distributed framework for emerging ai applications. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI'18, pp. 561-577, USA, 2018. USENIX Association. ISBN 9781931971478.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54771, May 2019. ISSN 0893-6080. doi: 10.1016 ht0.:///..mmmeet2211.......
Razvan Pascamu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1310 1318, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https: https::/prroceedngs..ttt http://weddnngg.....rrrres//e/ee wrw.preessseeee
http//////paaaaa133/ wttp://w...eeeaa////aaaaaiiiaaaa html
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashroe Anandkumar. Fourcastnct: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022. URL https://arxiv. http://aaxvvvo.rraabbaa/ otg/abs//202..1111 11214
Michael Penwarden, Shandian Zhe, Akil Narayan, and Robert M Kirby. A metalcarning approach for Physice-Informed neural networks (PINNs): Application to parameterized PDFs. J. Comput. Phys., 477 (11191)) March 2023.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-7000 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blci (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1030--138,, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceeddnn mlr. wtt.aiinn.mirr...es///a. wtt.vv/eeeeeeee11ee html
Jessie Richter-Powell, Yaron Lipman, and Ricky T. Q. Chen. Neural conservation laws: A divergence- free perspective. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 68075-38083 Cur- Associates, Inc., 2022. URL httpps:/prrcceee/// www.:eewwww...eenen.oeeo.eee.eee nwwrips www..nws....pppp..eeeeee caappr_fflesseeeeeeeeeppsspssese /tlee//paper//00///1e///
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 645 660, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-23788-6
Robin M. Schmidt. Recurrent neural networks (rnns): A gentle introduction and overview, 2019. URL
https://arxiv http:///aaxii..orraa/////../ wwtp://////.........55 1991/ 05911
16

---

Published in Transactions on Machine Learning Research (01/2026)
Andrea Soltoggio, Escoghene Ben-Iwhiwhu, Vladimir Braverman, Eric Eaton, Benjamin Epstein, Yunhao Ge, Lucy Halperin, Jonathan How, Laurent Ttti, Michael A. Jacobs, Pavan Kantharaju, Long Le, Steven Lee, Xinran Liu, Sildomar T. Monteiro, David Musliner, Saptarshi Nath, Priyadarshini Panda, Christos Peridis, Hamed Pirsiavash, Vishwa Parekh, Kaushik Roy, Shahaf Shperberg, Hava T. Siegelmann, Peter Stone, Kyle Vedder, Jingfeng Wu, Lin Yang, Guangyao Zheng, and Soheil Kolouri. A collective AI via lifelong learning and sharing at the edge. Nature Machine. Intellignnce 6(3):251-26:) March 2024. ISSN 2522-5839. doi: 10. 10.1038/s422 0008//44256-0224088000
Hwijae Son, Jin Woo Jang, Woo Jin Han, and Hyung Ju Hwang. Sobolev training for physics informed neural networks, 2021. URL https: /wwxxv.xvvvva///////a////aal wwweiii.rrr//eeo wwgg/bg///////e.eo 08932
Suraj Srinivas and Francois Fleuret.
Knowledge transfer with Jacobian matching. In Jennifer Dy and Andreas Krause (ods.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 4723 1731. PMLR, 10 10-15 Jul 2018. URL https; //proceedings... mlr. www.ppressss... htt.:///nniiaaaaaairraaa html
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xum Wang. Surrogate modeling for fluid flows based on plysiee-connttraieed deep learning without simulation data. Comput. Methods Appl. Mech. Eng., 361 (112732):112732, April 2020.
Fabricio Arend Torres, Marcello Massimo Negri, Marco Inversi, Jonathan Aellen, and Volker Roth. Lagrangian flow networks for conservation laws. In The Twelfth International Conference on Learning Representations, 2024. URL htttp:://openreview net/ nett/EorrrPidccshaahyyiiii
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ihhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 272, 2020. doi: 10.1035//445592-0194
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informen neural networks. SIAM J. Sci. Comput., 43(5)::3055-ABB)) A3081, Jamuary 2021.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics- neural networks, 2022a. URL https://arxiv.cr wwwp://////aaaae/eeeo /ttpaab...2222222/
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent
kernel perspective. Journal of Computational Physics, 449:110768, 2022b. ISSN 0021-9991. doi: 10tp:///10.00111/....22220 pi1/50021999221066xxx
URL https: http://wwwscciinedeeee http://wwcaaccccoto.o..cooo//aaa
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning From Multiple Experts: Self-paced Knowledge Distillation for Long- Tailed Classification. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 247 263, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58558-7, doi: 10.1007 10.1007/978-22 3-33--8558---7777777
Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on Knowledge Distillation for Large Language Models: Methods, Exaluation, and Application. ACM Thans. Intell. Syst. Technol., October 2024. ISSN 2157-6904. doi: 10.1145/3699518.
Xuan Zhang, Jacob Helwig, Yuchao Lin, Yaochen Xie, Cong Fu, Stephan Wojtowytsch, and Shuiwang Ji. Sinenct: Learning temporal dynamics in time-dependent partial differential equations. In The Twelfth International Conference on Learning Represenlations, 2024. URL http://ppppnrrriee.eeetttttoouee
LSYnE2hLVG
Colby Wight & Jia Zhao. Solving Allen- Cahn and Hahn-Hilliard equations using the adaptive physics informed neural networks. Commun. Comput. Phys., 29(3):930 -544, June 2021.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Zongren Zou and George Em Karniadakis. L-hydra: Multi-head physics-informed neural networks. ArXiv, abs/2301 02152, 2023. URL https: https://api cemontic
orr.aattccccol.......o.orrraaaaa wwt.:/ooo/rooss.255555 2554440033
18

---

Published in Transactions on Machine Learning Research (01/2026)
A Implementation Details and Setups
In this Section, give additional details the implementation of the models and their respective losses. We start by defining the individual loss components in the most general case. Each experiment will feature its required terms based on the problem definition.
Sobolev norms. Let u(æ:) Ω R be the function we want to learn. In the case of taking values in RD, can take the sum over the vector components. The main loss terms are those linked to learning the solution in the domain ft The distance in the Sobolev space Wm,.( between the function and our Neural Network & is defined as
||u all ( 1|u alll (1))
| D D'
(9)
where D'u is the I-th order differential of i.e. the gradient or Jacobian for 1 = 1 and the Hessian for = 2. Practically, these squared norms are approximated via Mean Squared Error (MSE) on a dataset Da of collocation points x; E Ω2 i = ,, Na with the respective values of the function and its differentials:
Da { Du(z)),
Du(z:),... .....
,'u((,,)))) 2t E D i= i..,..., No.
(10)
The MSE then calculated as
Na (u(æ;) - a(x;))²
Iu - a| 21/2(00)
Na
(11)
D'u- DVu-D'alleau D'all2
Na (D'u(z;)-D'a((r)))
-
1= 1=1,..,,
Each of the models will feature DERL, HESL and our other methodologies use only the second, with one more values of 1 + 0, and SOB uses both. In the case of deeeddepe problems, it is sufficient to consider the augmented domain i = T] x Ω and collocation points that are time-domain couples (t,, æz). 1, 1,...,,, Nu- The modols' derivatives are calculated using Automatic Differentiation (Baydin et al. 2018 with the python Py Torch package (Paszke al. 2019) and, in particular, its functions torch. func. jounc.jacra jacrev and torch. func. hessian. In the case of SOB, the derivative terms of order > 2 are approximated using discrete difference expectation over random vectors (see Czarnecki et al. (2017 and Rifai et al. 2011 for more details).
more terms in equation 11 In particular, OUTL uses only the first,
PDE residuals. For the evaluation of the physical consistency of a model and OUTL+PINN, we adopt the 1.2 norm of the MLPs PDE residual, as in Raissi et al. (2019). Let the problem be defined by Partial Differential Equation Flu(x)) O, where F is differential operator
Ou (x)(z)
Ja Ox, Ox,
F[u(x)]= = f(x) (z)
Σ
(x:
Σ
asj (x)
(x) + ...
(12)
Dz
and f(x) is function independent of u, usually called (external) forcing. The PDE residual measure we consider is the 1 norm of Fli), which is approximated again as the MSE over collocation points dataset DD defined above. In this case, there is no supervised target and the operator is applied to the model using AD, obtaining the loss term
Na Σ (F((())))
(13)
This loss will be used measure the consistency of our model to the physics of the problem or the PDE in general in strong sense since contains the derivatives of the network itself.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Boundary (and Initial) conditions. To learn the function u(x), per theorem 3.2 need to provide
some information at the boundary an as well. Otherwise, we can only conclude that ((x)) and ((::) differ by a possibly non-zero constant. In one dimension, this reminds us of the indefinite integral F(x) = f(z) + which is defined up to a constant C' determined by some condition on the function integral F. To measure the 1?2 distance between the true function and our approximation on the boundary, we employ the usual I, (822) norm approximated via MSE loss on a dataset of collocation points
D3 (12 b(x,),, 0, E EN,
i= I.....,, No.
(14)
where b(x:) is function defining the boundary conditions. The norm is then approximated as
N:
||u U (300)
Σ (b(2,)
(b())))(()))).
(15)
Na
In the case of time-dependent problems, we consider the augmented domain á = [0, x Q. Hore, the collocation points of the boundary conditions are time-space couples (4, @x.) E [0, T) x 0n. For time-dependent problems, initial conditions u(0, x) = g(x) are also required as part of the extended boundary and are learned with
N. Σ (g(2x) a(0, 2.))
[|u(0, .²) - a00 a: (L)) = N₂
(16)
In theory, the augmented boundary should contain the values of uT a) at the final time T. However, PDE problems are usually defined without a final condition, and it is more interesting not to have one to see which methodology can propagate physical information better. For example, SOTA methods as PINN do not include them (Raissi et al. 2019). Therefore, we decided not to include them well, leading to harder experimental setup than our theoretical results. For example, the experimental results in Appendix C.2 where the PINN failed to propagate the solution from & = 0 to higher times, confirm the findings of [aag et al. (202aa) of PINNs being biased during training. However, DERL did not suffer from this issue.
Each of the above components, when present, will have its weight the total loss, which is one hyperparamcter to be tuned, Below we report details on the baseline models and their losses.
Details for the baseline models. The baselines used in the main text comprehend:
OUTL, that is supervised learning of the solution u.
PUNL+ a combination of supervised learning of u and the additional information provided by the PINN loss (Raissi al. 2019). This provides the model with information on the underlying PDE and should increase its physical consistency.
SOB, that is Sobolev learning, first introduced in Czarnecki et al. 2017 which considers the both the losses u and its derivalives, as in equation
We report here the losses used for cach methodology to have a clean comparison between them. For space reasons, we write the true norms instead of the MSE approximations described above. Each norm is in (?) or (10, x ?!) based on the setting.
Loss comparison.
OUTL ((u, ú) Ap Apllu a| + A,CC + ABBC
OUTL+ PINN :L(u, 0) Apl|u 1 61| >>||F X/IC + ABBC
DERL =Luu, à) = A -
+ BBBC
DER+ + HESL: :L(u, a) = Ap (IIDu - Dāl +
- D + X,IC + BBBC
(17)
HESL: :[(u, (i) = AnI[D² DD 2 + A,C + BB
SOB -[(u,) (i) Ap (Ilu a||2 Du Da >,IC + ABBC
SOB + HES :L(u, (.) = Ap (Ilu 0||2 |Du D2ll2 |[D2u-DZalP) + X, + ABBC
PINN -(u, a) = A + I + BC
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Computational time epoch for each experiment with the Adam optimizer. Alhen-Cahn results are for the experimental setup in Appendix [C.]] For NCL we reported the computational time of 100 steps due to the definition and length of one epoch as in Richter- et al. (2022).
Model
Pendulum
Allen-Cahn
Continuity
Kdv
NCL
OUTL+PINN
6.136
4.009
12.61
37.57
DERL
3.599
2.575
3.867
17.85
8.358
HESL
25.43
DERSHESL
28.75
OUTL
3.376 3.331
2.442
5.380
14.73
7.826
SOB
2.564
5.887
18.17
8.3365
SOB+HES
24.34
where BC and IC are respectively given by equation 15 and equation 16
A.1 Computational time
Although effective, optimization of higher-order derivatives of Neural Network can be costly in time and computational terms. On the other hand, the functional and the Automatic Differentiation framework of Pytorch (Paszke et al. 2019) allowed us to calculate such quantities with ease and in contained time. For a complete comparison, we calculated the time to perform one training epoch with the Adam optimizer for each task and model. Results are available in table 7 As expected, OUTL is the fastest having the most basic objective, while DERI, beats both SOB and OUTL++PINN, It seems clear that the more terms in equation 17 the longer the training time, as seen by adding the Hessian matrices in the KdV experiment. Here, the third-order derivatives have huge impact PINN training. On the other hand, the approximations employed in SOB to calculate the Hessian matrices did not significantly improve the computation time compared to our approach with Automatic Differentiation. For reference, all the experiments were performed on an NVIDIA H100 GPU.
A.2 Model tuning
For the OOD generalization experiments in Section 4.2 that is the two parametric versions of Allen-Cahn and the pendulum experiments, each model was tuned individually on the task to find the best hyperparameters. To do so, performed standard train/validee waaiii/iiiaaaiion/ooooo /test pipeline. This was done to extract the best performance, to have fair comparison among the methodologies, and to measure their effectiveness in the tasks. The hyperparameters to be tuned are given by the weights of the different norms in equation 17 as well as the learning rate. The batch size was fixed for each experiment beforchand.
The tuning was conducted using the Ray library (Moritz al. 2018) with the HyperOpt search algorithm (Bergstra et al. 2013) and the ASHA scheduler (Li et al. (2020a) for early stopping of unpromising samples. For each weight AD,Ar,Ag the search space was the interval [10-3 10 with log-uniform distribution, and for the learning rate, we consider discrete values in [0.0001, 0.0005, 0.001]. Most importantly, the target metric of the tuning to evaluate individual runs was the L loss on the function ||u û (2()), being the usual target in data-driven tasks.
For the pendulum experiments that involve full trajectories, 30 trajectories are for training and 10 for validation, while 10 unscen trajectories are used for model testing. For the pendulum interpolation task in Section C.4. a total of 40 trajectories are used for in the tuning phase: points with times between (= 3.75 and 6.25 are unseen by the models and used for tuning/ tuning/validatiin validation, while 10 unseen complete trajectories are used for testing. For the parametric. Allen-Cahn experiment with a single parameter, we calculated the solution on a set of 1000 collocation points for 13 values of & E []; 8 are used for training, 2 for validation and 5 for iesting. Similarly, for the experiment with two parameters, we sampled 25 couples of (EI, E (0, 1]² with a 60/20/20 split.
Finally, a tuning procedure was applied to find the optimal weights for the experiments in Section 4.1 trained with the Adam optimizer. In these cases, no information on the test set is used and weights are found by
21

---

Published in Transactions on Machine Learning Research (01/2026)
optimizing the error solely training points, as it is in common practice for physical systems. This ensures there is no information leakage from the test set.
We remark that no tuning is performed for the experiments with distillation and physical knowledge transfer.
Proofs of the Theoretical Statements
B.1 Proof of theorem
Proof. Since u(t) ([o TD,, u is actually Holder continuous or, be precise, has a Holder continuous representative in the space (Maz'ya, 20111. Since L(u, ù) + O, we have that a'(t) L2(0,211, u'(t) and a(0) - u() (the initial conditions just one point). If v - û, have that v' 12, 0 and converges to function a.c. constant, which we can suppose to be continuous as above. Then, since v(0) = O, we have that = 0 and à u.
B.2 Proof of theorem 3.2
We begin by stating Poincaré inequality (see also Evans 2022 and Maz ya 20111).
Theorem B.1 [Evans] (2022) Section 5.6, theorem 3 and Section 5.8.1, theorem 1). Let S p < oo and S2 be subset bounded in at least one direction. Then, there erists a constant C, depending only on Q and p, such that for every function of the Sobolev space W of functions null at the boundary, it holds:
[| S Dullle)))))
(18)
In case the function is not necessarily rull at the boundary u E W.22(s)) and Q is bounded, we have that
||u (w)lllep0)) I Cl|Duullze0))
(19)
where (w)a =
1 [o] Ja u dr the average of u.
To prove one of our results we will also need the following generalization of Poincaré's inequality. Theorem B.2 (Maz' (2011) Section 6.11.1, corollary 2). Let Q RR be open set with finâte volume, WL)((S) such that the trace of u on the boundary 82 is r-integrable, that is [[ < co. Then, for every T,P.q such that (n - p)r S p(n -1 1) and rn a-1 it holds
| C (IIDullerra) + [[ulle)(0)))
(20)
In particular, p= 2 we have that q == but since Q has finâte volume, the inequality holds for each U 2n and, in particular, for = 2. m-1
2n n-1 and, since Ω has finite volume, it holds for Corollary B.3. If = 2, theorem B. holds for
each q a-1 and, in particular, for q 2, that is
|| s (( + [Illl
(21)
We are now ready prove theorem 3.2
Proof. Let ún be sequence such that L(ūns u) S en with e, + 0 and vn in -u. From the definition of L have that [[De, (La I en and similarly for ||.,]l22(00), that.
( s C (l) (aaaca + [[ualle((a)) 5 2Cen + -00
(22)
[v,.llii2(0)) ||w,ll ( [|Du,, (12 2(C + 1)c, +
which gives us the first part of the thesis. Additionally, Dr, + 0. This means that the limit of v, is a.e. constant and, actually, continuuusly differentiable in Q. Since v, + and lim, Do, = Doso (limits and weak derivatives commute), at the limit ag = u and, by continuity of the trace operator in W!)((2), we also
have that lag ulag in (82). The result can be casily extended to multi-component functions by considering each component individually.
22

---

Published in Transactions on Machine Learning Research (01/2026)
B.3 Proof of theorem 3.3
Proof. We show the results for time-independent PDEs; for time-dependent PDEs, it is sullicient. to consider the extended domain f = [) x Q2 in which the imitial (and final) conditions become boundary conditions. We also show the result for functions with onc component. The result can be generalizod by applying it to each component.
Let be the trained Neural network. From theorem 3.2 we have that a network à trained with the loss equation 6 with derivative terms converges to the solution of the PDE in Wm)) This implies that, when L [ūu
ü u) 0, all the derivatives of â converge to the true ones D' D'u for I I= O...., m. Since F is cont innuis, it follows that F(ū) + Flu] 0. As a consequence, DERL learns to fulfill the underlying PDE of the system.
Remark B.4. In the case of time-dependent PDEs, we will not provide the final conditions at ( t= T, which are part of the 'boundary' of the extended domain and are necessary for Theorem 3.3. Since our experiments are on regular functions, this has proven not to be an issuc, and the neural network still converges.
B.4 Proof of theorem 3.6
Proof. Assuming the result for smooth integrable functions, we show the thesis using the characterization of the Sobolev space (R"((R") via Ce (RR) approximations with smooth functions with compact support [Maz'ya ya, )011). For each > O, there exists ø E C2 (IR") such that ||u - @llwe))(e)) < 6. First, we note that
for each u E (RR"
[r¹ Ou (a + hiee;) dd
|u(x + hei) u(xx)
dr:
(23)
Ou
< 0 Oz,
(xx + hte;) dt/
so that, by squaring and integrating
u(x + he,) u(x:) 1
I 1.6(1
Ou Ôr:
Ou
(x hte,)) dt
< 12(()))
(24)
h
(,((R))
L2(D")
Oz,
which is the first part of the thesis. Applying this Φ one directly shows that
|D u D2 01 S liu.. 02.1le3(R))
(25)
Then, we have that
DD u Uz (z,1122((2)
[D u Dr o Dr, Φ- dz. 0z - -ux,
DD u D (122 + ||D o 2.. (alleR(R)) + 110.. un (a()))))
(26)
Mrr, &n. (12(3)))))) + + S 38
where the second inequality follows from equation 25 and the last one follows from choosing small enough The same steps are true for every open subset Ω by considering the Lioc norm.
C Additional Results and Experiments for Section 4
In this Section, we provide additional results and experiments. For the complete list of tasks table
C.1 Allen-Cahn equation
We now present the additional results for the Allnn-Cahn Cahn equation. Figure ba shows direct comparison of the errors the learned solution in the domain while figure 6b shows the error difference for the PDE
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8 Summary of the tasks.
Experiment
Equation
Description
Tasks
Generalize in-doraan
Generalize to new parameters
Allen-Cahn
(AC) M(uee Myn) uuuu"
Second-order PDE
Transfer to larger domain
Transfer to new parameters
Continuity
(CO) 8u + V (wx)
Time-dependent PDE
Generalize in-domain
Pendulum
(DP) à 4 sin(u)
|
A m
it=
ODE
Gencralize to new parameters
KdV distillation
(KdV) u uny VMlar
Third-order PDE
Higher-order distillation
(NS.M) [Du] 12 - Au Vp =
Navier-Stokes
System of PDEs (Appendix)
Generalize in-domain
(NS.I)V = 0
(A2.C) Up
V (pr)
Øt
Distillation different architecture
NCL distillation
(A2.IJV au
System PDEs (Appendix)
Vp
(A2.M)
Dulu
OUTL- DERL
OUN DERL
508 DERL
OUTL- DEAL
OUNL DERL
DERL
D.L55:
5.01551
C.0201
05
0 0.003)
05
2.00
2,0002
0.
6 000073
00
20002
-05
5.00244
-05
-10
50
-05
00
05
10
13
-65
00
05
10
-55
02
05
10
0.00555
-10
20
-55
60
05
10
10-10
-35
05
05
10
50
65
50
05
100
100003
(a)
(b)
Figure 6: Allen-Cahn experiment: (a) u error comparison between DERL and the other methodologics. (b) PDE residual comparison between DERL and the other methodologies. Blue regions are where DERL performs better.
residual. We computed the local error on grid points for each method. Then, for each method, we computed the difference between its errors and DERL's error, such that positive values (blue) are where DERL performs better. These pictures show clearly the performance of DERL and the improvements learning a physical system with its partial derivatives.
equispaced grid. We also experimented with learning from equispaced grid of collocation points with da = 0.02 and the ADAM optimizer (Kingma 2014p. Table g reports the metrics for this setup. For a graphical comparison, in figure we report the L error on u and PDE residual in the form of differences between the other methodologies and DERL: blue regions are where we perform better. The key takeaway is, as Section 4.1 that learning the derivatives both sufficient and better than learning just u, even with this setup. and SOB performed best in all metries, while OUTL and its combination with PINN at least 2 times worse.
Experiment
Table 9: Results for the Allen-Cahn equation grid points with Adam optimizer.
Model
||u á||a
|F[(|]|22
DERL (ours)
0.01235
0.01173
OUTL
0.03258
0.02157
OUTL+PINN
0.02109
0.02896
SOB
0.01500
0.01578
24

---

grid with Published in Transactions on Machine Learning Research (01/2020)
1,3
OUTL DERL
OUTL+PINN DERL
508 DERL
1.
CUTL DERL
OUTL-PINN DERL
50B DERL
305168
2A1122
3 0.212
1.1128
1A1357
-6.
8 00020
-6.
502000
103587
-L05603
-603799
10
65
12
05
65
05
05
10
35
80 05
1.0
(a) 1|u
2//2
(b) PDE residual
Figure 7: Graphical results for the Allen-Cahn experiment with grid points and Adam optimizer. I.2 loss on u and PDE residual comparison between the methodologies. Differences between the methodology and the DERL errors. Positive (blue) regions are where we perform better.
Time:
Time:
Time
05
6.3
351M
303
-35
25
-33
-55
-26
-35
10
25
-5
-12
-65
26
15
20
-5
65
25
10
15
Figure 8: Solution to the continuity equation at / 0,5,9
C.2 Continuity equation
Setup. The original solution was calculated using finite volumes (Ferziger & Peric 2001 Ar Ay 0.01 in the domain [-1.5, 1.5] and time discretization of At 0.001 for ł E used during training, we downsampled the full grid by a factor of 10 for each spatiotemporal coordinate. In the testing phase, we test the model on the full grid, without downsampling. This way, we can test which model can best learn the solution on a finer grid, while being consistent with the equation. The empirical derivatives are calculated with = 0.01 on each component.
0, 10 For the points
We start by showing the true solution the continuity equation I errors t 0 (figure Dab, t t= (figure 9b), and best methodologies to learn the densily are DERL and OUTL, with the latter being the worst at physical consistency as reported in Section 4.1 For this task, we performed an additional experiment with PINN. We found similar results to Wang et al. (2022a), where the model deviates from the true solution as time increases. We notice that DERL, while not having access to the density at later times, does not show such behavior.
0, 0,5,9 figure together with the (figure Dch. The results show clearly that the
Randomly sampled points. For this equation, we performed an additional experiment where we tested the effects of using an interpolated curve on the solution to randomly sample points and calculate derivatives. This simulates a situation with sparse data and, as above, no analytic derivatives u available. For this purpose, we used the data from the grid calculated as in Section 4.1 and used a regular grid interpolator from SciPy (Virtanen al. 2020) with cubic interpolation for third-order accuracy. Then, we randomly sampled 10000 points in the 1, domain, and calculated finite difference derivatives with h 0.001.
Numerical results are available in table 10 while figures 10b and 10a show the comparisons physical consistency (PDE 2 residual) and the 2 error on P across methodologics. The format is the usual difference between the hods' error and the DERL one, with positive values (blue regions) meaning we are performing better. Even this case DERL among the best-perf models, with a u error and physical consistency respectively similar SOB. OUTL is clearly the worst learning the physics of the problem, even combination with the PINN loss. The PINN alone shows again discrepancy from the true solution that increases with time. This shows that our methodology works even with empirical derivatives calculated
25

---

Published in Transactions on Machine Learning Research (01/2026)
5.8866
1.0
5.0752
0.5
0.0564
-35
1,2
Error t
0
6:1541
1.5
0.3370
0.5
0.392
-2.5
1,2
(b) Error at l 5
OUTLIPIINN
S08
PINN
0.267
1.5
0.1555
0.1524
0.332
-2.5
(c) Error al =
Figure 9: Continuity equation experiment.
(a) True densities at 0,5, 9. L errors for the compared
methodologies at (b) f = 0, (c) t = 5, (d) t t= 9.
CUTL-CERL
DELL
SCB- DCRL
ITR 2014
OUTL- CER.
OUTLIFRN-EEN
SCB DELL
ITA 2011
833
-18
-12
ACTIT
(a) ||u a 2|2
(b)
||
Figure 10: Continuity equation task with randomly sampled points. Differences between the other methods' residuals and DERL at f 5. Blue regions are where we perform better.
Table 10: Numerical results for the continuity equation with randomly sampled points.
Model
||M 412
|[[
DERL (ours)
0.08714
0.1948
OUTL
0.07308
0.4065
OUTL+PINN
0.1390
0.4393
SOB
0.07190
0.1815
PINN
0.3296
0.1499
26

---

Published in Transactions on Machine Learning Research (01/2026)
OUTL-
OUTL+PINN- -ERL
506 DERL
2004
0,012
0,000
0.008
03
0.3
0.006
6.
3.30
21.0
1.5
city
OUTL vorticty
vortictty
1.0
1.0
1.0
0.2
03
9.2 -10
-0.5
0.0
0.5
1.0
-3.
-4.
0.0
0.5
1.0
-1.
-2.
8.0
0.
1.4
0.0
Figure 11: Error difference plot for the Kovasznay flow experiment on the Navier-Stokes equation. The first row is the local error the solution (u, p), while the second row is the error the vorticity w. Each point is calculated as the difference between the error by the model and the error by DERL. Blue regions are where DERL performs better.
interpolation of the true solution. We remark again the importance of learning the derivatives, DERL and SOB are the best all-around models for this task.
C.3 Navier-Stokes Kovasznay flow
The analytical solution to the Kovasznay flow task is
A ex sin sin(2ry) u(I,y)= xx cos(2
p(z, v) (1 - 22r).
1
(27)
2n
where \
4.2 + 4x2. In this case, the vorticity is given by Am Figure 11 shows the error difference between DERL and the other methodologies, both for the solution (u, p) and the vorticity w.
C.4 Pendulum
We start by describing the pendulum problem from a physical point of view. We consider the rope pendulum under the force of gravity Fg = - and a dampening force Fa = bbū where u(t) the angle the rope makes w.r.t. the vertical, rope and the mass of the pendulum. A schematic representation the system can be found in figure 12a and
parameter for the dampening force, m are respectively the length of the
also plot its phase space on (u, û) along with some trajectories with dampening 12b without 12c that is the conservative case. The evolution of the pendulum is given by the ODE F[u] mlu + mg sin(u) + blū 0
C.4.1 Differentiability with respect to Initial Conditions
For these kinds of ODEs, the solution (u(t, ug, úu), útt up ún)) is actually continuously differentiable with respect to the IC (uo, úo). For a proof, see Harlman (2002), theorem 3.1. In this specific case, we can derive the following PDEs:
① 8u
g
0 sin(u) Ouo
b a du m Oug Øt b a du m Büg Øt
0,
Duo 0t
1
Glu]
(28)
8
g
@sin(u)
Dig 0t2
1
Dûo
27

---

Published in Transactions on Machine Learning Research (01/2026)
Phase space vajectones
Phase space trjectorics
(())
01-
0.5
0.
Fy
mg sinis)
my co)()())
mu
0.0
(a)
(b)
(c)
Figure 12: (a) Schematic representation of the pendulum system: the state variables are the angle u the rope makes with the vertical direction. (b) Phase space the damped pendulum. Arrows represent the direction and intensity of ú, w. Testing trajectories are also plotted. (c) Phase space of the conservative pendulum with dampening (b 0).
which are true for any trajectory (u(t, ugs úo); ti(t, uor úg)). In the experiments, we calculated the L PDE residual of equation 28 in plain PINN style, to which model learns to be differentiable w.r.t. the IC.
C.4.2 HNN and LNN setup
In this Soction, we describe the setup we used for the Hamiltonian Neural Network (Greydanus et al., 2019 and Lagrangian Neural Network (Cranmer al., 2019). In particular, will explain how we adapted these methods to learn non-conservative fields, which not present in their original works.
In both the conservative and dampened cases, we used the setup described in their original articles without modifications the architecture, initialization, and training. We now describe how to include the dampening effects through data or particular processing during inference.
Hamiltonian Neural Network. In this case, the neural network parametrizes the Hamiltonian function Hp,.)), from which derivatives one obtains
0p
0H
0g Ōt
0H 0p
(29)
Ot
0q
Equation 29 is valid in the conservative case, while in the dampened case it is sufficient to add the term bb to öp de leading to the correct formulation as in equation (DP) in table g This way, the HNN learns just the conservative part of the field, which was originally made for, while we add the dampening contribution externally.
Lagrangian Neural Network. In this case, need to look at the Lagrangian formulation of mechanics, in particular at the Buler-Lagrangian equation. Given neural network that parametrizes the Lagrangian function L(u, ú), we have that (see Cranmer et al. (2019) for the full explanation)
dt D&C D,L= 0.
(30)
We can add the effect of external forces, in this case, the dampening, using d' Aleembert's principle of virtual work [Goldstein] 2011). In particular, given the force in vector form
Fa = (ii(cos(()) (u)] and the position of the pendulum function of the coordinates
(31)
x1) [1 sin(u). -I cos(w))
(32)
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Results for the damped pendulum experiment with randomly sampled times and empirical derivatives.
Model
||x
á||2
||
ú||2
|[F[0]]|2
|||úū]|
DERL (ours)
0.01237
0.01504
0.01581
0.1106
0.4678
OUTL
0.01.499
0.07637
0.07148
0.2466
1.001
OUTL+PINN
0.01495
0.06993
0.06588
0,2877
0.8437
SOB
0.01532
0.02983
0.02890
0.1276
0.8365
Table 12: Results for the conservative pendulum experiment.
Model
||M - & la
|| - ål 2
|
||]|0]|2
[= ll
DERL (ours)
0.07079
0.08459
0.04233
0.35976
0.5577
OUTL
0.1066
0.2856
0.2510
1.865
0.8998
OUTL+PINN
0.07638
0.1643
0.1296
0.7740
0.8422
SOB
0.06707
0.1031
0.06986
0.5090
0.7970
the virtual work is given by
Q u, ú) Fa dx(()
-bl
(33)
Ou
Tt then sufficient put this term on the right-hand side of equation 30 and proceed with the calculations as in Cranmer al. (2019) to obtain the full update of the model.
C.4.3 Results with empirical derivatives
We repeated the same experiment of Section this time using empirical derivatives calculated by using finite difference with h = 10-3 We also used randomly sampled times to evaluate trajectories, to simulate a setting where the time sampling not constant. We report the relevant losses in table [1 where DERL outperforms the other methods every metric, especially those showing the generalization capability. We conclude that learning derivatives improves generalization the models while achicving SOTA results predicting the trajectories.
C.4.4 Conservative pendulum
The setup for the conservative pendulum experiment is the same as for the dampened one, with the fundamental difference that
in both data generation and during training. We report the numerical
results in table 12 with the same definitions as in Section 4.2 We also plot the field error differences in figure DERL the best model in the conservative case well, since achieves the lowest error in almost all metries. The only exception is the error on testing trajectories, where it is still second best and very close to SOB. In general, other method shows the generalization capabilities of DERL, especially in terms of PDE residuals and regularity with respect to initial conditions.
In this case, HNN (fiel L error 0. 0.13570) and LNN (fiel L error 0.085388) putperform all other methodologies as they are precisely built for this task. We still remark that their objective is solely to learn the field, which is easier than whole trajectories.
C.5 Parametric Allen-Cahn Equation
We provide additional information on the Allen-Cahn parametric experiments, together with additional results.
C.5.1 Experiment with One parameter
Setup. The MLPs have 4 hidden layers with 50 units each, similar to Raissi al. (019)) The models take input the coordinate and parameter of the equation (ar, y, E) and output the solution for the Allen-Cahn
29

---

Published in Transactions on Machine Learning Research (01/2020)
OUTL- DERL
OUTL+PINN DERL
SOB DERL
DERL
LNN
-0.0008
1
0 0.0828
-01664
0.3385
2501
0,208
0
0.413
0.0950
-0.0107
0 5,509
0.5855
-0.1204
-1
1
-02301
0.7518
-0.398
(a)
(b)
Figure 13: Comparison in the learned field at = 0 for the pendulum experiment expressed as the differences
of the L errors between the other methodologies and DERL. The blue area is where we perform is better.
DERL
OUTL
OUTL+PINN
SOB
10
10
PDE residual
10-2
3
1
2
3
E
E
Figure 14: Allen Cahn parametric experiments: Single parameter equation. u error (left) and PDE residual (right) as functions of g.
equation with parameter &, that 0(x E). The networks are trained with batch size 100 with the Adam optimizer for 50 epochs, then are further optimized for another 50 epochs with the BFGS optimizer on the whole dataset, similarly to Penwarden et al. (2023). The solution calculated for 16 values of &. A veaiin/viiaaiion////ooo /test split is applied, with 8 parameters for training, 2 for validation, and 6 for testing. Testing values of & are chosen to explore regions of the paramcter space not seen during training and validation. The training and validation datasets are used for tuning the hyperparameters of the models, that is the learning rate and the weights of the loss terms in equation
1000 random collocation points is
Results. Numerical results are provided in Section 4.2 Figure 14 shows the error on and the PDE residual of cach model for different values of the parameters. As we can see, DERL is consistently the best model. We also provide the comparison plots for the solution u on the testing 5 values in figure [5] As before, the plots show a comparison between DERL and the other methods' errors, such that blue regions are where DERL performs better. It is clear also from visual point of view that DERL is the best model to predict u and to simultancously have low PDE residual, especially compared to OUTL.
C.5.2 Experiment with Two Parameters
This Section provides additional results and information on the sctup for the double paramctric Allen-Cahh experiment in Section 4.2 The MLPs have hidden layers with 50 units each, similar to Raissi el al. (2019). The models take as input the coordinate and parameters of the equation (æ,y, &1. E2) and output the estimate of with parameters (2), that â(x, y: E2). The networks are trained with batch size 100 with the
Adam optimizer for 100 epochs, then are further optimized for another 100 epochs with the BFGS optimizer the whole dataset, similarly to Penwarden et. al. (2023). The solution 1000 random collocation points
30

---

Published in Transactions on Machine Learning Research (01/2026)
OUTI
OUT -PINN DERL
DEAL
OUTL DEAL
OUTLIFINN DERL
S0B DERL
0.004005
8L3004500
02004576
0.5
000366
8.2002406
0000379
D00774
-65
0,2007566
-0 0001520
-03
0.003655
3,006022
-1.
1.0
355
50
05
34
5.0
-5
6.0
65
24
-55
60 65
0001
1.
-05
05
1.0
-1-00
-03
05
63
2.0
-83
62
68 12
(a) error for &
0.6
(b) PDE residual for = 0.6
3.0
DEAL
DERL
DERL
A.0001534
6305225
4 A.0093112
1000816
6353225
200
6 A.0007718
0.0001106
2003114
1.
-33
05
05
1.0
-25
-65
00
35
30
-.0
405
0M
55
1.5
(c) u error for E 0.9
(d) PDE residual
= 0.9
2.0
OUTI DDBL
QUT, PINN DERI
SCB DERL
1.0
OUTL- DEAL
OUTL PINN DERL
506 DERL
6306715
RL0005200
6.05971
25
0.5
6.324430
5.
6 4.0003345
0.0
0.009.915
6.32255
-15
-05
-1.0
-65
00
05
13-14
-55
06
65
14-14 -55 02
- 000000022
-35
02
05
50
-25
-05
05
6.30017
(e) u error for { = 1.4
(f) PDE residual for = 1.4
FINN DERL
A.00091
6357030
0.0094775
6354637
A 0000023
0.0003766
05
(g) a error for 2.3
(h) PDE residual for {
3.0
OUTL- DERL
OUTLFRIN DERL
S0B DERL
6.0009224
1.0
-UTL-DFRI
OUTL -PIN-INN-DP
S0B DERL
5.3369
3.5
A.501216
0.5
5A3579
6.0040333
2.048300
3.0
0.0002510
0.0
5002036
01001733
195
-25
-05
0,0000012
-3.
-63
00
03
L--13
-5
05
03
-1-
-5
U5 03
-1.0
10
05
00
05
1.
-10
-55
0.0
05
3.5
-30
05
00
05 3,0
---300028
(i) u error for & = 2.6
(j) PDE residual for & = 2.6
OUTL+RIN
OUTL
-ERL
0.5003322
20011
8C007225
AC00017C
0.2577
A
00000013
0.20330
0.000006
0.0014
20
(k) error & 2.8
(1) PDE residual for 2.8
Figure 15: Comparison plots for the solution and the foreing term in the Allen-Cahn parametric equation with one parameter on test values of g
31

---

Published in Transactions on Machine Learning Research (01/2026)
DERL
OUTL
OUTL+PINN
S0B
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
&
E1
Figure 16: Double-paramette Allen-Cahn equation. The plot is obtained by taking grid of points E in the parameter space and coloring it with the color of the model with the lowest error or PDE residual for such parameters.
is calculated for 25 couples of (L1, E2). A train/validdation/ten split is applied, with 15 couples of parameters for training, for validation, and 5 for testing. The training and validation datasets are used for tuning the hyperparameters of the models, that the learning rate and the weights of the loss terms in equation 17
Figure 16 shows the parameters' space colored with a color code representing the best model in that region. In particular, each point in the plot represents couple of parameters, which are colored with the color of the model achieving the lowest. error and PDE residual for that couple of parameters. DERL the best in the whole domain.
Similarly to before, figure 17 shows the error comparisons on u and the PDE residuals for the 5 testing couples (2). Figures clearly show that DERL outperforms the other models.
D Additional Results and Experiments for Section 5
D.1 Higher Order Derivatives
Teacher and student model setup. We consider the Korteweg-de Vries (KdV) equation, a third-order non-linear PDE (equation (KdV) in table 8), with 0.0025. The 1C is u(0,z)) cos(wx) with periodic BC for and Mg. The reference solution is first obtained using the Scipy (Virtanen et al. 2020) solver with Fast Fourier Transforms
grid with Ar At 0.005. Then, PINN is carefully trained on equation (KdV) in table and is treated as the teacher model to distill knowledge from. For the teacher model, we used an MLP with 9 layers of 50 units, tanh activation, trained for 200 epochs with batch size 64 and Adam optimizer. The higher number layers is applied due to the complexity of the KdV task, as in Jagtap et al, (2020). After the teacher's training, we save a dataset hessian matrix D² (2x, t) on the training collocation points. As always, partial derivatives are calculated via AD Baydin et al. (2018). Then, the student networks are trained with the specific distillation loss of the method, e.g. Dûs Dūr for DERL, and the original BC and IC. Since this is third-order PDE, interested in understanding the impact of higher-order derivatives on the performance, both in terms of the PDE residual and in terms of matching the true solution u or the BC. We implemented Hessian learning (HESL), which learns the Hessian matrices D²år the teacher model. We also explored its combination with DERL HESL, which learns both Dår and D2 t)) and Soboley (SOB + HESL.. In this last case, we use the approximation of the Hessian matrix as suggested in Czarnecki al. (2017). while for HESL the full Hessian of the network, the cost. of increase in computational time (see Section A.1 for
outputs @r(x, t). derivatives Dür (2c, 8), and
also
32

---

Published in Transactions on Machine Learning Research (01/2026)
OUTL DEAL
OUTL& DEAL
SDB DEAL
OUTL DERL
OUTL+ANN DERL
50B DERL
301367
000354
100666
B3529
00052
62552
0.02255
6.411
0:0:11
62210
-321.
-102237
0.0201
(a) for (0.10475525, 0.64767021]
(b) PDE residual for [0.10475525, 0.64767021]
3.8 OUTL DERL
OUTL+PINN DEAL
50B DEAL
0.005751
6:005
0.
000336
0.00356
63005
0.3
0.003300
0.
0:001400
-82
6.360
0.300
-0 03802020
-3.2
65
65
14
18
60
85
00002745
-15
6.0
25
10
5.
0.0209
(c) error for (0.26152518, 0.46585182]
(d) PDE residual for [0.26152518, 0.46585182
DERL
OUTL+FINN DERL
50B
OUTL DERL
OUTL+ANN DERL
50B DERL
6.2236
8207546
0.002477
6.462
0.004455
5.222
000341
63351
6.5
DO13T7
-12
14
45
35
00003666
-.0020
(e) for 00.42562175, 0.51911434]
PDE residual for (0.42562175, 0.51911434]
OUTL OUTL+RIN DERL
2.1166
02013
502510
63759
207765
6.257
0.3
203525
63000
201196
-0.
-83
6 62252
5A1812
-10
10
25
20
05
05
50
-55
7o-16
-63
(g) error for (0.61230216, 0.31101022]
(h) PDE residual for [0.61230216, 0.31101022)
OUTL DERL
OUTL+PINN DERL
50B DERL
1.
OUTL-DERL
GUTL+
S08 DEAL
63580
8 801256
30055
20000
B1256
65
-101555
-30162
D2205
(i) error for (0.6681 8045, 0.99318447)
(i) PDE residual for [0.66818045, 0.9931 8447]
Figure 17: Comparison plots for the solution and the forcing term in the Allen-Cahn parametric equation with two parameters on test couples (2)
33

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: Results for the KdV equation, PINN distillation. Results are empirical L norms over the time-space domain. Derivative and Hessian losses are computed with respect the PINN. Best modcl(s) in bold, second best underlined.
Model
Iis
ulla
[Dis Dürllz
[[Pār Daull2
|[[ús]]l2
BC loss
PINN (teacher)
0.037171
0.16638
0.33532
DERL (ours)
0.038331
0,098188
3,9872
0.32480
0.014197
HESL (ours)
0.037380
0.065454
1.1662
0.19153
0.014220
DER+ HESL (ours)
0.038524
0.041988
0.85280
0.19317
0.031850
OUTL
0.038589
0.22580
31.582
17.366
0.012830
SOB
0.037447
0.10097
4.0967
0.38523
0.013644
SOB+HES
0.041353
0.13119
3,2684
0.23184
0.016222
Table 14: Results for the NCL distillation experiment. L distance between the NCL and our distilled
solutions, 2 norms of the residuals of the 2 PDEs. Norms are calculated across the entire time-space domain. The best results are in bold.
Model
1 usll
(A2.M) |[ (A2.1) |F(as]l|
DERL (ours)
0.015287
0.28566
0.095044
OUTL
0.022247
0.52101
0.17159
SOB
0.013282
0,28620
0.10134
further details). Each methodology has been tuned individually to find the hyperparameters that provide the best approximation of the true teacher's solution ê7.
Results. Table 13 shows the relevant metrics. Together with the usual ones, we also report the I distances between the teacher's s partial derivatives and the student's ones, that is ||Dr Dūsllz and ||D - D² s and the BC loss. First, we notice that all meth successfully learned to approximate the solution u with the same performance as the teacher model, with SOB+HESL being the worst by a small margin. We observe that (a) distilling a PINN can lead noticcable performance improvements. In particular, our HESL and DER+HESL achieved the same PDE residual and distance to the true solution of the teacher model but with BC loss smaller by at least one order of magnitude. This may be due to the easier optimization objective of the student models. (b) OUTL fails to distill the physical knowledge of architecturally identical PINN, the PDE residual is two orders of magnitude larger than any other model. Interestingly, the best methods in terms of physical consistencies are those that did not see the values of u directly. (c) When boundary conditions are available for both u, ugs second order derivatives are sufficient to learn the true solution with PDE consistency comparable to PINN. (d) HESL and D showed the best overall results. This tells us that adding one more derivative helps learning and distilling high-order PDEs. Furthermore, approximating Hessians in SOB+HEST (Czarnecki et al. 2017 slightly reduces the performance compared to using the full Hessian of the network.
Figures 18a and 18b show on the (t,r) domain the comparison of the errors on u(t, x) and the local PDE residual for the tested distillation mothods.
As we can see, every lested method learns similarly to approximate the true solution w(4, x),, with slightly better performances on models with fewer derivatives learn. On the other hand, more derivatives help the consistency of the model to the underlying PDE compared to no derivatives, as in OUTL, which fails in this regard. It is worth noticing that the PINN not able to optimize the BC, while other methodologies adapted well to them.
D.2 Neural Conservation Laws
We perform knowledge distillation with the NCL architecture RRichter-Powell et al. 2022). considering the Euler equations for incompressible inviscid fluids with variable density in the 3D unit ball (3 PDEs). The system made of the mass conservation (A2.C), incompressibility (A2.T), and momentum (A2.M) equations
34

---

Published in Transactions on Machine Learning Research (01/2026)
DERL
OUTL
SOB
DERL
OUTL
SOB
1.0
0.5
0.5
0.0
0.09391
1.837
0.0835
1.633
1.0
0.06666
-1.0
1.224
SOB
PINN
HESL
0.0522:
SOB
PINN
HESL
1.020
1.0
1.0
0.0313:
0.612
0. 0.02081
0.408
4.204
0.00001
0.000
0.5
0.5
-1.0
110
1.00.0
1.0
0.5
1.00.0
0.5
(a)
(b)
Figure 18: Korteweg-de de Vries equation experiment. (a) Errors on the true solution u in the (t, x) space. (b)
Local PDE L2 residual. In OUTL, the residual capped at 2 for clarity reasons on the other plots.
(table where is the density, u the velocity and p the pressure. IC and BC are from Richter-Powell et al. (2022) Equation (A2.D) is guaranteed by the specific architecture the NCL model, while (A2.M) and
(A2.1) are learned with the usual PINN style PDE residual loss. The teacher and student models' setup the same as in Richter-Powell al. 2022).
Model setup. The model architecture the same as in Richter--Pwell et al. (2022), that a MLP with 4 layers of 128 units, trained for 10000 steps on batches of 1000 random points the 3D unit ball. As in the previous task, each student model has the same architecture the teacher model, and the hyperparameters are tuned for the best loss imitating the teacher's output. Each student model is trained for 10 epochs the dataset created by the output and derivatives of the teacher network. We used more than 1 epoch to ensure convergence of the models. We remark that the errors on the solution are calculated with respect to the NCL model, as no numerical solver we tried converged to the true solution.
Results. As Equation (A2.D) is guaranteed by NCL design, table 14 reports results for Equations (A2.M) and (A2.1). Although minimal, distillation reported an improvement in the BC (from 0.09 for the teacher to around 0.07 for the students). DERL and SOB outperform OUTL, with almost 50% error reduction each metric. Even with a different architecture like NCT., these results highlight the effectiveness of derivative distillation for the transfer of physical knowledge across models.
Figure 19 shows the comparison among methodologies in terms of error differences between OUTL, SOB, and DERL that positive blue regions are where our methodology performs better. We plot the errors its Dās with respect to the NCL teacher model on the Z = 0 plane, as well as the PDE residual crror on the momentum (A2.M) and incompressibilitty (A2.I) equations in table Except for some initial conditions, DERL outperforms the other two methodologies, especially in the u, Du errors and in the momentum equation consistency.
D.3 Transfer to Larger Timespan
Setup. The timespan extension task conducted shows how DERL can be leveraged to improve existing physical model by extending its working time domain. DERL allows for the transfer of physical information from teacher model to a student one that is currently training to solve an equation new times. The equation of interest. is the KdV equation (KdV) in table S The architecture of each model consists 9 layers with 50 units each and tanh activation. this case, no tuning is performed and the parameters are ABC 0.001, èrc 10, APDE 1., which provided optimal results with the PINN trained on the whole domain. We simulate a setting where the solution is be found with no prior knowledge, interested in the model that best works out-of-the-box. The full time-space domain is [0, [-1, 1],
35

---

Published in Transactions on Machine Learning Research (01/2020)
AJS
11
(a) | üs 222 comparison
(b) |āār Dås comparison
(c) (A2.M) [[[(a]]l2 comparison
(d) (A2.T) [[F[0)]]l comparison
Figure 19: Neural Conservation Laws distillation. Comparisons between the methodologies on u, Du errors w.r.t. the reference model and PDE residuals on the Z plane. All plots represent the difference between the method error and the DERL error that positive (blue) regions where DERL performs better.
which we divide in three time intervals: the first [0, 0.35] [-1, 1], the middle interval [0.35, 0.7] x [-1, and the final [0.35, 1] x [-1,1]. The training procedures consists of three main steps and here provide an in-depth description of them.
PINN is trained for 30.000 steps with 1000 random collocations points each with the Adam optimizer in the first interval [o, 0.35) [-1,1]. Once the PINN is trained, we evaluate its output â and partial derivatives Då (calculated via AD (Baydin al. 2018)) on the collocation points and save them in dataset.
2. A second model is trained for 30.000 steps and the Adam optimizer with the same loss as in equation that consists of (a) PINN loss term to find the solution to the equation in the second interval [0.35, 0.7] [-1, 1], (b) distillation loss on the dataset from step 1, that is on points of the region [0, 0.35) x [-1, 1]. This specific loss depends on the method used for distillation, which can be either DERL, OUTL, or SOB. (c) The initial condition loss (c). The boundary condition loss for the now extended region [0, 0.7] x [-1, Tn this step, we consider two possibilities: we load the weights from the model of step 1, which we indicate with continual, we start from a randomly initialized network, which we indicate with from-scralch. Finally, we consider model that loads the weights of step 1 but has no distillation term for the lower-left quadrant and should forget that region: call this model PINN with no distillation. Either way, we consider the final model to be predictor for the solution u in the cunulative interval [0, 0.7] x [-1, 1]. We again save a dataset of its evaluation on the collocation points of the lower half the domain.
[-1, and using the PINN loss on the remaining interval [0.7, x [-1,1], together with the BC loss on the whole region and the IC loss. We keep the continual and from-scraach pipelines separated: a model that starts from scratch in stage 2 will start. from scratch again, and vice versa, model that was fine-tuned in stage 2 will be
We repeat step 2, this time by distilling information from [0, 0.7]
36

---

Published in Transactions on Machine Learning Research (01/2020)
DC1L fram wculdh
DElL comsred
ScB brmnarrahh
50B continal
OuT, Frmm strath
OUT' cortinue
NWW ro didl.
3380
3337
10
-1
Figure 20: Prediction error in the timo-space domain for the transforring experiment larger timespans
Prediction Error
10a
frorm-srratch
SOB continual
continual
DERL continual
OUTL from scratch
PINN full
PINN with replay
SOB from-scratch
10 -1
Loss
10.2
10:3
20000
40000
60000
80000
100000
Training Step
Figure 21: Prediction errors for the KdV transfer experiment. Vertical lines represent when a new step of incremental training initiated.
fine-tuned in this stage as well. The final model used predict the solution in the entire domain [0, x [-1,1].
For a final comparison, we considered a PINN that is trained at once for the total sum of 100.000 steps in the full domain [0, 1] x [-1, 1], as this represents a model that is trained not incrementally.
Results. We now provide additional results for the transferring experiment on larger timespans Section 5.1 Figure 20 shows the prediction error in the full time-space domain (t, E [0. [-1, We clearly see how DERI. outperforms the model, especially compared to OUTL and the standard PINN, which converges early to suboptimal solution. Finally, Figure 21 shows the prediction errors the whole domain function of the training steps for all the methods. We see that the results of from-scratch are similar to the ones of the continual approach, with the latter performing slightly better.
D.4 Transfer new domain
Setup. The domain extension task we conducted shows how DERL can be leveraged to improve an existing physical model by extending working domain. DERL allows for the transfer of physical information from a teacher model to a student one that is currently training to solve an equation in new region of the domain. This way, the student model should be good predictor of the solution of the equation in larger domain than its teacher. The equation of interest is the Allen-Cahn equation (AC) in table g The architecture of the model is the same as in Scction 4.1 that is we always work with MLPs with 4 layers of 50 neurons, similar to [Riisi et al. 2019). In this case, no tuning is performed. We simulate a setting where the solution is to be found with no prior knowledge, as we are interested in the model that best works out-oof-the-bee box, without any particular training procodure. The full domain is [-1, 1] [-1, which we divide in the lower-left quadrant
[1. x [], the lower-right quadrant [o, 1] x [-1,0] and the upper section [-1,1] [o, 1] as in figure
We first sampled total of 1000 random collocation points in the full domain. We also calculated the
37

---

Published in Transactions on Machine Learning Research (01/2026)
Error of the predicted solution
1.0
0.40
0.35
0.5
0.30
0.25
0.0
0.20
0.15
-0.5
0.10
0.05
-1.0
0.00
-1.0
-0.5
0.5
1.0
Figure 22: Error on u for the PINN model with no distillation.
boundary conditions for each sub-region and the full one. Our experiment consists of three main steps and here we provide an in-depth description of them.
1. A PINN is trained for 100 epochs with the BFGS optimizer in the lower-left quadrant with the corresponding random collocation points and boundary conditions. Once the PINN is trained, we evaluate its output and partial derivatives Då (calculated via AD (Baydin et al. 2018 on the collocation points and save them in a dataset.
A second model is trained for 100 epochs and the BFGS optimizer with the same loss equation that consists of (a) PINN loss term to find the solution to the equation in the lower-right, quadrant. [0 x [-1, O], (b) distillation loss on the dataset from step 1, that is points of the region 0] x [-1, 0]. This specific loss depends on the method used for distillation which can be cither DERL, OUTL, or SOB. (c) The boundary condition loss for the new extended region [-1, [-1, 0j. In this step, we consider two possibilities: we load the weights from the model of step 1, which we indicate with continual, or we start from a randomly initialized network, which we indicate with from-scratch. Finally, we consider model that loads the weights of step but has no distillation term for the lower-left quadrant and should forget that region: call this model PINN with distillation. Either way, consider the final model to be predictor for the solution in the lower half of the domain. We again save dataset of its evaluation on the collocation points of the lower half of the domain.
We repeat step 2, this time by distilling the lower half of the domain [-1, x [-1,0] and using the PINN loss on the upper half of the domain [-1, x [0, 1], together with the BC loss on the whole region. We keep the continual and from-scratch pipelines separated: model that starts from scratch in stage 2 will start from scratch again, and vice versa, model that was fine-tuned in stage 2 will be fine-tuned in this stage well. The final model is used to predict the solution in the entire domain [-1.1] x [1,
For a final comparison, we considered PINN that trained at once for the total sum of 300 epochs in the full domain [-1, [-1,1], as this represents a model that trained not incrementally.
Results. As seen in Section 5.2, the PINN model with no distillation performs poorly and forgets the information from the lower half of the domain, which can be noticed from the error in figure
22 We plot the
error comparison of the other methodologies and their PDE residuals in figures 23 and 24
38

---

Published in Transactions on Machine Learning Research (01/2026)
OUTL DERL
DERL
PINN Tull-DERL DERL
3.0
OUTL DERL
5OB-DER
PINN full- DERL
0.0 3054012
08-
0.000 2003826
51400
2022039
51123
3.0
0000000
5.0
50866
-05
3. 2220220
2.201207
2004
562
5 5.09263
9.203980
10 -23
-05
00
(a) error comparison
(b) PDE residual comparison
Figure 23: Comparison between DERL and the other models for the domain extension experiment with fine-tuning.
1.0 OUTL DERL
SOB DEAL
PINN Full- DERL
0.31112
1.0
DUTL DERL
S0B S0B-DEL DERL
PINN full DERL
3:21216
3.5554
3 335753
0.0
3.20172
3,0000
330011
-05
3.03204
-05
-300333
-0 03362
-6 2201355
000058
-0.03533
(a) error comparison
(b) PDE residual comparison
Figure 24: Comparison between DERL and the other models for the domain extension experiment starting from scratch at each step.
D.5 Transfer new parameters
Setup. We consider the Allen-Cahn equation (AC) table S in its parametric form with two parameters. The parametric component is again f, which is calculatod from the analytical solution
u(x, E2,, = Σ sin sin(jay)
j2
The architecture of the model is the same as in Section 4.1 that we always work with MLPPs with layers of 50 neurons, similarly to Raissi al. (2019). The models take input (I, &1, and predict a(æ, y: (2). In this case, no luning is performed. We simulate a setting where the solution to be found with no prior knowledge, particular procedure. The domain is [-1,1] x [1,, and we consider a total of 25 couples of parameters (2), 10 are used in the first step, 10 in the second step, and the rest for final testing purposes. We sample a total of 1000 random collocation points which will be used in every step, together with all couples of parameters. Our experiment consists of two main steps:
interested in the model that best works tit--fffthh-bx.. without any
PINN trained for 100 epochs with the Adam optimizer and another 100 with the BFGS optimizer for all the combinations of the collocation points and the first 10 values of &1, Once the PINN is trained, we evaluate its output á and partial derivatives Då (calculated via AD (Baydin et al. 2018 on the collocation points and save them in datasct.
2. A second model is trained for 100 epochs with the Adam optimizer and another 100 with the BFGS optimizer with loss that consists of (a) PINN loss term to find the solution to the parametric equation for the next 10 couples of &1, (b) distillation loss on the datasct from step 1, that is on previous values of the parameters. This specific loss depends on the method used for dist illaiona,oo which can be either DERL, OUTL, or SOB. (c) The boundary condition loss. We again consider two possibilitics: we load the weights from the model of step 1, which we indicate with continual, or we start from randomly initialized network, which we indicate with fronchcratch. Other possibilities in D.4 We finally test the models the last couple of parameters.
39

---

Published in Transactions on Machine Learning Research (01/2026)
DERL
OUTL
SOB
PINN full
PINN no dist.
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
E1
Figure 25: Allen-Cahn parameter transfer experiment with distillation on a new model from scratch. Each square represents the best model with the lowest error or PDE residual in the &i parameter space.
Results. In this case, the PINN with no distillation does not fall behind too much as it seems to maintain good generalization capabilities among the paramcters.
E On the Effect of Noise and Step Size on DERL
In this Section, experiment with empirical derivatives with different values for the step size h and with white noise applied to the true derivative values, to simulate the effects of real measurements on DERL. We aim to show the robustness of DERL under realistic conditions. However, step size is not an issue in our experimental setup: as shown in Section 4.1 and in Appendix C.4.3, one can apply a cubic interpolation on the available data to calculate empirical derivatives with any step size. Furthermore, step sizes and noise are never an issue in the transfer learning setup of Section where partial derivatives are always calculated on the teacher network, without any need for approximation.
We experiment with the Allen-Cahn equation in Section 4.1 for the in-domaín gemeralization, and with the damped Pendulum experiment in Section 4.2 for the generalization to new parameters. We experiment with the step sizes h = 0.001, 0.01, 0.05 and with Gaussian noise with a 0.001, 0.01, 0.05. We also report the results SOB and OUTL+PINN (only for the noise, as empirical derivatives are not used) with the same setup for a direct comparison of the two methods.
Results for the Allen-Cahn experiment experiment available in Table 16 As can see, DERL is robust to different noise values and with increasing step size. We notice that similar patterns observed in DERL are also present with SOB and OUTL+PINN, which the least robust. by far, showing that DERL extracts all the available information from partial derivatives, even without accessing information on u directly.
available in Table 15 while results for the damped pendulum
40

---

Published in Transactions on Machine Learning Research (01/2026)
Table 15: Results with different step size h and noise a for the Allen- Cahn equation. the prediction and the 2 PDE residual.
We include the L error
Model
u 0||2
x10
[[ x x10
Varying stop size h
DERI à 0.001
0.7654
2.097
SOB h 0.001
0.8733
2.173
DERL 0.01
0.6769
2.071
SOB 0.01
1.034
2.556
DERL 0.05
4.904
4.226
SOB h = 0.05
4.504
4.339
Varying noise a
DERL a = 0.001
0.6250
1.870
SOB = 0.001
1.497
3.589
OUTL PINN = 0.001
9.176
14.97
DERL = 0.01
1.398
2.430
SOB 0.01
1.854
4.007
OUTL+PINN 0.01
8.537
13.45
DERL 0.05
3.726
6.321
SOB 0.05
4.235
5.717
OUTL+PINN 0.05
19.06
15.57
Table 16: Results with different step size and noise for the damped pendulum experiments equation. We include the same metrics as in Section 4.2 for the testing trajectorics.
Model
Iu 0112
||
A|I2 ||F[0]l|2
|
|| t= ||
Varying step size h
DERL h = 0.001
0.01237
0.01504
0.01581
0.1106
0.4678
SOB h = 0.001
0.01670
0.02894
0.03018
0.1240
0.6704
DERL 0.01
0.02002
0.01445
0.02161
0.07980
0.3565
SOB h = 0.01
0.01726
0.02912
0.02919
0.1245
0.6627
DERL 0.05
0.0574
0.06421
0.07272
0.2617
0.9187
SOB h 0.05
0.02927
0.04066
0.04015
0.1396
0.6785
Varying noise
DERL 0.001
0.01536
0.01540
0.01678
0.09847
0.4440
SOB 0.001
0.008728
0.03116
0.02898
0.1482
0.6346
DUTI+PINN 0.001
0.009781
0.05475
0.05515
0.2272
0.7717
DERL = 0.01
0.02486
0.04159
0.03421
0.1872
0.5949
SOB σ = 0.01
0.02203
0.04326
0.04245
0.2063
0.8317
PUTL+PINN = 0.01
0.02317
0.07735
0.07343
0.3511
0.8831
DERL σ = 0.05
0.05432
0.09773
0.09407
0.4848
0.7811
SOB a 0.05
0.04768
0.1098
0.1073
0.7025
1.652
OUTL+PINN 0.05
0.05277
0.1848
0.1769
1.054
1.540
41


Reviews:
[{"review_id": "8aaLBCnfqj", "reviewer": "Reviewer_fGEP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes DERL, a supervised learning framework that leverages derivative information to model physical systems and introduces a derivative-based distillation protocol to enable incremental learning. The authors demonstrate that by distilling derivatives rather than just output values, the model can effectively transfer knowledge across different time horizons, spatial domains, and physical parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is well-organized and supported by extensive experimental validation across various ODE and PDE tasks, showing clear improvements over standard baselines.", "audience_yn": "Yes", "audience_explanation": "Machine Learning (ML) techniques have found great success in modeling dynamical and physical systems, including Partial Differential Equations (PDEs). This topic has attracted a great number of researchers. The paper focus on a derivative learning framework that can model physical systems by learning their partial derivatives, which also outperforms state-of-the-art methods in generalizing an ODE to unseen initial conditions and a parametric PDE to unseen parameters. I believe that the TMLR's audience will be interested in the findings in this paper.", "requested_changes": "1. Justification of Distillation vs. Experience Replay: The authors frame the problem as incremental learning using distillation. However, it is unclear why a distillation-based approach is necessary compared to a simple Experience Replay baseline. Since the analytical derivatives (or empirical ones) are readily available, a straightforward alternative would be to retain a small buffer of old data points and train the model jointly with the new data. The paper needs to justify why we should trust a 'Teacher' model's compressed knowledge over the 'Raw Data's' ground truth. A comparison with a replay-based baseline would significantly strengthen the motivation for the proposed distillation protocol.\n\n2. Incomplete Robustness Analysis (Missing PINN Comparison): The proposed method's reliance on 'empirical derivatives' computed via finite differences raises concerns about its robustness, a critical issue for real-world applications where measurement data is often noisy. While I acknowledge the sensitivity analysis provided in Appendix F, it is insufficient as it only demonstrates DERL's performance degradation in isolation (or compared to SOB). Crucially, the analysis omits a comparison against a standard PINN baseline under the same noisy conditions. This comparison is essential: a PINN treats the PDE residual as a soft regularization term, which typically offers inherent robustness against high-frequency noise. In contrast, DERL's supervised framework fits the model's gradients directly to noisy derivative targets, potentially causing it to overfit to high-frequency artifacts (noise amplification). To properly assess the method's practical utility, the authors should provide a comparative study demonstrating whether DERL maintains its advantage over PINNs in noisy regimes.\n\n3. Asymmetric Baseline Comparison in Incremental Learning: The authors compare Incremental DERL against Joint Training PINN (\"PINN full\"), but omit the critical \"DERL full\" baseline (Joint Training on the entire domain using the DERL loss). Since Section 4 has already established that the DERL loss formulation is superior to the PINN loss, the current comparison in Section 5 conflates the benefits of the loss function with the effects of the training strategy. Without a \"DERL full\" baseline, it is impossible to isolate the impact of the incremental strategy or explicitly quantify the performance trade-off (e.g., error accumulation) incurred by using distillation instead of joint training. The authors should include this baseline to disentangle these factors.", "broader_impact_concerns": "Physics-informed neural networks has become a promising tool in solving PDEs via deep learning. The paper extends the working domain of a PINN and to generalize better across different PDE parameterizations. The work also shows promising intersections with continual learning, as DERL’s distillation allowed us to build physical models incrementally, showing minimal to no forgetting of previous knowledge.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ECG2OxXTU5", "reviewer": "Reviewer_6y99", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Derivative Learning (DERL), a supervised method for learning physical systems by training a neural network to match partial derivatives of the target function rather than its direct values. The central idea is that, for ODEs and PDEs, the evolution of a physical system is fully determined by its derivatives, along with the initial and boundary conditions. \n\nStrength:\n1. The author evaluated multiple ODEs and PDEs for experiments, e.g., Allen–Cahn, Navier-Stokes, etc. It provides evidence that the proposed method works for solving differential equations; however, it requires knowledge of the partial derivatives. \n2. The proposed method can be transferred between equations with different parameters, which means the proposed method not only learns the parameter of the PDE itself, but also the underlying system. This is an interesting point.\n3. The idea of using partial derivatives to train the model and solve the PDE is interesting (Even though it is not practical, see the weakness, please).\n\nWeakness:\n1. The primary concern I have is that the proposed method requires the partial derivatives as the input data, e.g., it is known before the equation is solved. From my limited knowledge, it is not feasible for many PDE systems. This means we often see the function u rather than its derivative when solving the equation.  For instance, when we solve the heat equation, we can observe the temperature of a point in the domain at a given time, but we do not know the derivatives of the temperature with respect to time at that point. \n2. As the author mentioned in the experiment setting, the partial derivatives used for training the model are computed from the solution or approximated by the numerical method, e.g., finite difference. For a fair comparison, it would be necessary to have the baseline for the solution from the traditional numerical method directly. Also, the baseline for the conventional numerical methods should be the one with strong performance, e.g., the finite element method, since it does not require an additional step to compute the partial derivatives.\n3. The different solution for the different PDEs seems to use different neural network architectures. It looks like the author reports results from the selected hyperparameters without explaining the selection protocol or the candidate settings.", "claims_and_evidence_yn": "No", "claims_explanation": "Please see my points in the weakness section above.", "audience_yn": "Yes", "audience_explanation": "Using neural networks to solve differential equations is an interesting topic; it is related to AI for science, which is a critical domain from my point of view.", "requested_changes": "Please address the point in the weakness section, thanks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DwGzelMsqM", "reviewer": "Reviewer_khjG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a framework for using neural networks to learn the evolution of physical systems, represented by differential equations. Rather than traditional methods, the proposed method, DERL, is trained on partial derivatives and can therefore achieve better performance in both theoretical domains, and is validated by empirical experiments. Besides, generalizability is also showcased via distillation for the DERL model. Experiments show that compared with other methods, the DERL framework can learn the model with higher accuracy and can be generalized well to unseen domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide theoretical analysis about why learning derivatives can provide enough information for learning the solution of the differential equations. Apart from that, several empirical experiments are carried out and according to the reported results, the error terms relative to the baselines are indeed lower, which implies that the framework of DERL can achieve better performance than the baselines, including supervised learning and the PINN method. Therefore, the claim made by the authors is established.", "audience_yn": "Yes", "audience_explanation": "The topic of this work is using learning-based methods to solve complex differential equations, which is pervasive in the real world. The complexity of the real-world dynamical system naturally creates challenges for learning, and the results can, inversely provide insights for the machine learning community.", "requested_changes": "1. In this work, one important part is the establishment of the cost function. In equation (6) the derivative information is used to perform supervised learning. However, in the real world, especially in a system that is dominated by an unknown DE, such a derivative \"evaluation\" can be hard to obtain. This is clearly a limitation of the work since, currently, the derivative is calculated by differentiation from rollouts. Although the authors mentioned that in the appendix section C, it would be more rigorous to put it in the main text to clearify the contribution of this work. \n\n2. It would be nice if the authors can discuss more about the comparison between the proposed DERL framework and PINN framework. Essentially, in the proposed method, the residuals are discarded from the cost function and the authors claimed that the derivative information alone is enough (together with boundaries) to recover the full solution. This is pretty counterintuitive. Although the authors invoked the results about the Sobolev space, it would be clearer to see such a discussion explicitly as remarks or lemmas. \n\n3. Fig 5 is visually hard to parse since the dotted lines and the other lines are entangled. It would be better if the author can present the illustration in a clearer way.", "broader_impact_concerns": "No ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)