Title: Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: PDEs

Neural Operators

Neural Emulators

Physics

Surrogate

Optimization
Abstract: Neural operators or emulators for PDEs trained on data from numerical solvers are conventionally assumed to be limited by their training data's fidelity. We challenge this assumption by identifying "emulator superiority," where neural networks trained purely on low-fidelity solver data can achieve higher accuracy than those solvers when evaluated against a higher-fidelity reference. Our theoretical analysis reveals how the interplay between emulator inductive biases, training objectives, and numerical error characteristics enables superior performance during multi-step rollouts. We empirically validate this finding across different PDEs using standard neural architectures, demonstrating that emulators can implicitly learn dynamics that are more regularized or exhibit more favorable error accumulation properties than their training data, potentially surpassing training data limitations and mitigating numerical artifacts. This work prompts a re-evaluation of emulator benchmarking, suggesting neural emulators might achieve greater physical fidelity than their training source within specific operational regimes.

Project Page: https://tum-pbs.github.io/emulator-superiority


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and investigates \"emulator superiority\". An interesting phenomenon where neural emulators trained solely on low-fidelity numerical solver data can outperform these solvers when evaluated against higher-fidelity references. Through theoretical analysis on linear PDEs, the authors demonstrate that certain neural architectures' inductive biases enable better error accumulation or generalization than their training solvers. Empirical experiments extend this concept to nonlinear PDEs, showing superiority across different architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- One of the strong aspects of this paper is its theoretical analysis. The authors provide clear and detailed proofs in the Fourier domain demonstrating conditions under which the \"emulator superiority\" can arise for linear PDEs.\n- The core concept of emulator superiority is interesting.\n\nWeaknesses:\n- Despite robust theoretical treatment for linear PDEs, the paper’s empirical experiments are somewhat limited to idealized, canonical PDE examples. While appropriate for proof-of-concept, the generalizability to real-world multi-physics simulations (e.g., climate, combustion, plasma physics) remains unexplored.\n- The paper sticks to one-step supervised training. It acknowledges but does not explore other paradigms like unrolled training or physics-informed losses, which might enhance or inhibit superiority. This makes me wonder to what extent is “superiority” an artifact of the chosen loss metric, test regime, or evaluation horizon? Would the observed superiority still exist if evaluated on alternative physics-informed metrics?\n- The superiority proofs rely on mode-isolated training and evaluation. For practical PDE problems, this may not be the case. Is the assumption of single-mode initial conditions meaningful in real scientific scenarios?\n\nQUESTIONS:\n- The paper notes that superiority may disappear when training data is already high-fidelity. Can we quantify a fidelity threshold below which emulators have room to improve?\n- If the core motivation of the work is to advocate for low-fidelity simulation-based emulators, it would be beneficial to include a discussion on the trade-offs in computational cost. Currently, there is limited analysis of the cost-benefit balance when using neural emulators, particularly considering that solving high-fidelity PDEs with neural models can involve significant training and inference costs, which may offset the claimed advantages.\n- How robust are the closed-form superiority proofs for linear PDEs to perturbations in discretization, boundary conditions, or higher-dimensional settings? Are these proofs extensible beyond 1D periodic domains and synthetic solvers?\n- Were baseline solvers used in training (e.g., Picard-truncated Burgers solver) intentionally made poor to create room for superiority? Could the superiority vanish if the low-fidelity solver were just marginally improved? \n- Can you try “non-toy” PDE systems (e.g., Navier-Stokes)?\n- To me, some of the training conditions (e.g., ICs from a single Fourier mode) are overly restrictive. How would the results change if training data sampled from more physically diverse regimes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors investigate when a learned PDE solver can be more accurate than the classical PDE solver used to generate training data.\n\nTheoretical analysis performed for three linear PDEs (convection, diffusion, stationary diffusion) and linear emulators revealed that learned solver can be more accurate if distributions of train and test data do not match.\n\nFor the nonlinear case the improved accuracy of learned solver was observed for nonlinear emulators (FNO, Dilated ResNet, Transformer, etc) on convection and Burgers equations for the case of identical distributions for train and test data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. Authors ask genuinely interesting question\n2. Both theoretical and empirical considerations are available\n\n**Weaknesses:**\n1. Both theoretical and empirical results are not convincing\n2. The setup authors consider is far from establish practices\n\nI provide more details in the section below.\n\nQUESTIONS:\n1. **Assumptions in the proof of theoretical results**\n\n   Authors assumed that the training data come from a very specific distribution that covers only a single mode. In this case it is clear (not explained explicitly) that the learned model will depend on the train set only from the mode number and both amplitude and phase are irrelevant. I have several questions on that:\n\n   a. Can the authors please supply missing proof that both amplitude and phase are irrelevant? I would suggest showing that equation (8) can be achieved when the usual loss function is considered, e.g., mean (with respect to distribution on phase and amplitude) $L_2$ error.\n\n   b. The whole formulation seems rather artificial with a weak dependence on training data. Can the authors please defend their choice of learning problem? Why is this simple scheme sufficient and interesting for operator learning?\n\n   c. Since authors work with linear models, it seems to be possible to obtain closed-form solutions for arbitrary data. Can the authors comment on the possible extension of their results on more realistic data distributions (e.g., more than one mode)?\n   \n2. **Theoretical results are trivial from the standpoint of backward error analysis**\n\n   Solution for PDE can be considered as an evaluation problem $y = \\phi(x)$, where $x$ is input data, $y$ is solution to PDE and $\\phi$ is an unknown map that brings input data to the solution. The approximate method that solves PDE is some other map $\\hat{\\phi}(x)$ that approximates $\\phi(x)$ sufficiently well. Standard forward error used by authors is $\\left\\|\\phi(x)-\\hat{\\phi}(x)\\right\\|$, where $\\left\\|\\cdot\\right\\|$ is a suitable norm.\n   \n   Backward error is based on an entirely different idea. Consider perturbation of input data $\\Delta x$ such that $\\hat{\\phi}(x) = \\phi(x + \\Delta x)$. In case this perturbation is possible to find $\\hat{\\phi}(x)$ can be considered as an *exact* solver for *wrong* input data. For solver $\\hat{\\phi}$ backward error is thus $\\left\\|\\Delta x\\right\\|$ (here $\\left\\|\\cdot\\right\\|$ is potentially different norm) and is related to forward error by condition number of the problem.\n   \n   If both $\\phi(x)$ and $\\hat{\\phi}(x)$ are continuous one may expect that one can find input $\\widetilde{x}$ near $x + \\Delta x$ such that $\\left\\|\\phi(\\widetilde{x}) - \\hat{\\phi}(\\widetilde{x})\\right\\| \\leq \\epsilon$ for small $\\epsilon$. Such inputs can be found by solving $\\arg\\min_{x}\\left\\|\\phi(x) - \\hat{\\phi}(x)\\right\\|$ (at least locally near $x$).\n   \n   Authors considered two different solution methods for the same problem $\\phi_1(x)$, $\\phi_2(x)$. These methods in general have different backward errors $\\Delta x_1(x)$ and $\\Delta x_2(x)$. Because of that these methods also have different solutions for the optimisation problem above $x_1 = \\arg\\min_{x}\\left\\|\\phi(x) - \\phi_1(x)\\right\\|\\neq x_2 = \\arg\\min_{x}\\left\\|\\phi(x) - \\phi_2(x)\\right\\|$.\n   \n   These observations do not look too interesting or surprising. And the main problem is we deliberately try to find inputs such that one method is superior to the other one. Several questions are in order:\n\n   a. Can the authors please comment on the considerations on backward error analysis?\n\n   b. On the higher level authors simply note that different methods are good on different inputs. This observation is well known in the literature on numerical methods. For example, trapezoidal rules have exponential convergence for periodic functions and are expected to give very small errors in this case, but for non-periodic functions the error will increase. Given that it is easy to find functions for which trapezoidal rule is superior to, say, Simpson's rule and vice versa. Can the authors explain what made their result conceptually different?\n\n   c. There is a clear parallel between adversarial examples for DL classification and inputs that lead to superiority. Did authors try to consider this parallel? What authors think in general on this relation?\n   \n3. **Empirical results for Burgers equation are not explained by theory**\n\n   In my view the results presented for Burgers equation is the most surprising one. It is surprising because superiority shows up in the same solution curve, still out of distribution but not completely unrelated to the training data.\n\n   a. It does not seem to me that theoretical results explain this behaviour. Can authors comment on whether a linear case is related to nonlinear one?\n\n   b. Can this result be replicated for other equations (e.g., KS, KdV) and architectures (e.g., FNO, ResNet)?\n\n   c. Do these results survive when one considers more diverse training sets?\n\nIn conclusion, I find questions raised by the author interesting and potentially fruitful for further study. Unfortunately, I do not find answers that authors provide to be useful or particularly interesting. Being said that I remain open to change my opinion after the rebuttal. I encourage authors to defend their position, supply more results and continue research on the topic.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper outlines the idea of emulator superiority, where a neural surrogate trained on low fidelity solver data is able to outperform said low-fidelity solver when evaluated on a high fidelity data. The paper also provides some theoretical analysis showing that a linear emulator in limited settings can have lower error than implicit and explicit numerical schemes in comparison to an analytical solution.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n1. The authors do a good job of formalizing their problem as well as describing the different types of generalization that are important for neural emulators applied to differential equations.\n2. The paper has useful theoretical results with closed form solutions for linear PDEs using a linear parametric ansatz.\n\n# Weaknesses:\n1. While the theoretical results for linear PDEs are good, the results for the nonlinear neural architectures are mixed, especially in the case of state-space superiority.\n\n2. The practical applicability of this paper is limited because of the problem setup. If a high fidelity simulation data is available, it is unlikely that a researcher would train a neural network with low fidelity data and then evaluate it on high fidelity data. More clarity on when exactly the proposed problem setting is valid is necessary.\n\nQUESTIONS:\n1. In the nonlinear PDE experiments, isn't it possible that the reason for neural network superiority on certain timescales is due to the ability of neural networks to smoothly interpolate, while the static low fidelity solver cannot do this?\n\n2. In the case of the advection equation, the narrative says \"Our theoretical analysis reveals several crucial conditions for achieving superiority. First, we must train a low-capacity linear ansatz on a high-capacity (non-analytical) reference method.\" How does this assumption fit with neural network architectures when applied to real data? It is often the case that modern neural networks are high capacity, so doesn't this mean that in many practical scenarios, neural network superiority will not be achieved?\n\n3. In Figure 1, there are labels called \"Amplifying\" and \"Dampening\". Does \"Amplifying\" mean that the scale of the solution is increasing or diverging? And does \"Dampening\" mean the solution scale is decreasing?\n\n4. For the Burger's equation experiment, it is stated that \"for this experiment, we train a vanilla UNet (Ronneberger et al., 2015) on only the initial \"smooth\" frames from this low-fidelity solver.\" Does that mean that the UNet is trained with a limited time-series rollout from the low fidelity solver and then during the test phase, it is made to do a longer rollout than it was trained with?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a surprising phenomenon in neural PDE modelling where a neural network solver trained only on low-fidelity numerical solver can outperform that solver in accuracy when evaluated against high-fidelity simulation results. The author conducts analysis on linear and non-linear PDEs with both linear and non-linear ansatz like UNet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The study of whether a neural solver trained on low-fidelity data can outperform the source is novel and important to scientific ML.\n\n2. The analysis for linear PDEs is clear and provides intuition for more complex non-linear cases. They show how superiority arises by analyzing linearized spectral errors and optimal filters. \n\nWeakness:\n\n1. The experiments and analysis are limited to 1D PDEs, which are far from the complexity of most real-world applications. The analysis for linear PDEs relies heavily on mode-wise error decomposition, which works for 1D linear PDEs due to the orthogonality and separability of modes. It remains unclear whether the analysis will extend to more realistic scenarios involving nonlinear, multi-dimensional PDEs, such as those found in fluid dynamics, where mode interactions, geometry are significantly more complex and high-fidelity simulations are expensive or difficult to obtain.\n\n2. The analysis of nonlinear neural architectures (e.g., UNets, Transformers, FNOs) is relatively scarce. It could be interesting to see more in-depth discussion regarding the relationship between effective receptive field of model and its rollout error / superiority.\n\nQUESTIONS:\nCan the authors provide a general intuition for what kinds of non-linear PDEs are more amenable to neural networks with limited receptive fields (e.g., feedforward ConvNets), and which ones may require larger context (e.g., long-range dependencies in turbulent flows or stiff PDEs)?\n\nLIMITATIONS:\nThe authors acknowledge one major limitation in the paper, the solver needs to be sufficiently “coarse” for neural emulator to exhibit superiority.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces emulator superiority which defines the phenomenon that machine learning models trained on low-fidelity solver may outperform the data-generating solver in terms of several metrics that measure autoregressive generalization, state-space generalization as defined in the paper. This emulator superiority is brought about by two primary sources: smaller accumulation error from multi-step rollouts and the inductive bias of the specific neural network used as an operator solver.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n- This paper clearly illustrates experimental motivations, preliminaries and experimental settings, helping readers to follow. The granularity of discussions seems well spread over the paper --- where main descriptions are in the manuscript and in-depth mathematical descriptions are appropriately deferred to appendix.\n\n- Motivations are well described and strong – that I agree the conventional wisdoms goes that the model is as good as the data (but not better). However, recent empirical findings suggest strong extrapolating capacities of neural networks especially in predicting high frequency modes given low-fidelity data, which calls for the analyses over the underlying reasons of such.\n\n- This paper seems novel in that it differs from prior approaches which explicitly trains a model to perform superresolution or extrapolation.\n\nWeaknesses: \n\n- Limited nonlinear problems: would bear larger contribution if the superiority is met across a more complex and harder prediction tasks.\n\n- While several potential error source are enumerated, such as spatial discretization errors, linearization errors, etc, it seems impossible to dissect exactly why and where this emulator superiority arises seems a daunting task. E.g., what inductive bias of ConvNet / ResNET / FNO / Transformer constitute for what portion of the superiority is hard to disentangle. In such case, the practical value of this work seems questionable.\n\nQUESTIONS:\n(1) Do any variations in the model contribute to higher superiority? Such as the kernel size or the number of modes in FNO. These experiments may provide guidance when solving a new problem not shown in this paper, such as Navier-Stokes.\n\n(2) Real-world observations often contain structured noise. Assuming we use such data rather than idealized solvers, what would this mean to this superiority?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}