Title: EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale
Venue: NeurIPS 2025
Accepted: yes
Keywords: ML for PDEs

fluid dynamics

turbulence

simulation

large-eddy simulation

Transformer

large-scale

spectral methods
Abstract: Computationally resolving turbulence remains a central challenge in fluid dynamics due to its multi-scale interactions. Fully resolving large-scale turbulence through direct numerical simulation (DNS) is computationally prohibitive, motivating data-driven machine learning alternatives. In this work, we propose EddyFormer, a Transformer-based spectral-element (SEM) architecture for large-scale turbulence simulation that combines the accuracy of spectral methods with the scalability of the attention mechanism. We introduce an SEM tokenization that decomposes the flow into grid-scale and subgrid-scale components, enabling capture of both local and global features. We create a new three-dimensional isotropic turbulence dataset and train EddyFormer to achieves DNS-level accuracy at 256^3 resolution, providing a 30x speedup over DNS. When applied to unseen domains up to 4x larger than in training, EddyFormer preserves accuracy on physics-invariant metrics-energy spectra, correlation functions, and structure functions-showing domain generalization. On The Well benchmark suite of diverse turbulent flows, EddyFormer resolves cases where prior ML models fail to converge, accurately reproducing complex dynamics across a wide range of physical conditions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for learned simulation of high-res turbulent flows. It combines a SEM representation, LES-style filtering and a specialized transformer variant. It claims higher accuracy on three experimental domains with 2D and 3D turbulence.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n\n3D turbulent flow simulation is quite challenging, and the results are impressive. In general, the paper is well written, provides a nice problem overview & experimental setting. It also performs a very decent evaluations on the experimental results from a CFD perspective, studying e.g. correlation and energy spectra.\n\n**Weaknesses**:\n\nWhile the paper seems thorough on the CFD side, it falls flat with its contributions to ML. \nFirst, the method description remains vague and at a high level, and it would likely be hard to replicate with the information provided (see questions below for specifics). \n\nNext, the architecture feels quite arbitrary (again see questions for details), and more importantly, the choices aren't ablated at all. So we don't know if any performance gains are due to SEM, or how much the SEMConv layers matter, etc.\nAnd finally, I'm quite concerned about whether baseline comparisons were performed properly. For example, Transolver claims superior performance on several CFD benchmarks, including to FNO-- but here it performs significantly worse than FNO. Worse, KF4 is almost identical to the dataset used in the Kochkov paper, yet in this paper Kochkov seems to perform 10x worse than a simple Resnet-- this can't be right.\n\nIt's possible there are certain properties of the domains studied here that cause these differences, or maybe evaluation in these previous works has issues. But this needs to be properly investigated, otherwise we end up with several papers all claiming SOTA with opposing results, and we learn nothing.\n\nSimilarly, the impact of the architectural choices needs to be studied and ablated. It's unlikely that someone will reimplement _exactly_ this method as it has a lot of moving pieces--but maybe SEM could be a useful representation for other problems? Or maybe LES splitting, or adding conv layers into a transform is actually very helpful for CFD? These are questions we'd want answered in order for others to build on this work.\n\nQUESTIONS:\n1. The method needs to be explained better:\n  - It sounds like the transformer operates on N^3 tokens, so what happens to the M^3 modes? Are they stacked into the feature dimensions? But also the latent width is specified as 32, so is there a latent encoding from M^3 -> 32 somewhere? What about SEMConv?The paper shouid clarify which operation act in which dimensions, and how those are transformed.\n  - If SEMConv also operates on N^3 tokens, why do we want a spectral convolution? The N^3 blocks are in real space, right?\n  - SEMConv in general needs a more detailed description (maybe in the appendix)-- padding, integration, etc.\n  - How is the model optimized? L2 loss on the output u?\n  - Which inputs and outputs are filtered, and how? (3) mentions SGS is unfiltered and LES is filtered, but then 3.1 suggests there are filters W_{in}, and W_{SGS} for SGS? \n2. Is there actually any advantage to splitting the model into a SGS, LES path? I understand that the architecture is inspired by LES solvers, but from a model perspective there's little actually enforcing these semantics. Loss is on the sum of both path, and after a layer or two information will be mixed in latents across both branches anyways. Have you tried just using one path (either with both filtered u_SGS and u_LES inputs, or just u) with more layers or higher capacity?\n3. Transformer with convolutional blocks is a bit of a weird combination. Can you just use a (bigger) standard transformer with FFNs, maybe with more blocks so that attention can actually do it's thing? Or if the SEMConv layers are very useful, do you need the transformer at all-- i.e. can you just run a FNO-style method on SEM elements?\n4. Besides the comments on strange baseline performance numbers, there also isn't any information on how the baselines are parametrized and run. Eg How many layers/width/... (ResNet and FNO can mean a lot of things). What resolution did you choose (you probably can't run a ResNet on 256^3)?  How was the field downsampled?\n5. It seems in general FNO performs only a bit worse than EF, but then it completely fails on RB/RT instabilities while EF performs very well. Why do you think that is?\n\nI think this paper really needs a careful revision, investigating baseline performance, and careful examination & ablation of the model architecture. This is probably not feasible to achieve within the rebuttal timeline. While I will increase my score on the off-chance that the authors can quickly clear up the comparisons, and add interesting findings re/ architecture, I'd actually encourage to spend a bit of time on this and resubmit a potentially much stronger & more impactful paper.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focus on studying neural operators on direct numerical simulation (DNS) dataset. The existing study of neural operators all tends to perform poorly on this challenging task due to the large-scale data (e.g. $256^3$ and $384^3$) and the complexity of the nature of turbulence.\n\nThe architecture of the proposed neural operator, EddyFormer, composes of two parts: SEMConv block and SEMAttn block. This design is inspired by the formulation of large eddy simulation (LES), where the solution is decomposed into two parts: LES stream and subgrid-scale (SGS) stream as equation (6). And the two streams are handled by the above blocks accordingly. Namely, SEMConv deals with local eddy dynamics of SGS stream, and SEMAttn handles long-range dependencies of LES stream (See Fig. 2).\n\nSuch heuristic design is proven to be highly advantageous with DNS datasets. Datasets are challenging that scale from $96^3$ to $256^3$ and compared with numerical solution on $384^3$ for homogeneous isotropic turbulence. Also, 2D turbulence model, Kolmogrov flow is considered. Finally, turbulence beyond homogeneous isotropic is benchmark in the Well dataset. In three benchmarks, EddyFormer surpasses all baselines by large gap.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe proposed method, EddyFormer, is deliberately designed with LES decomposition, which could be the reason for outperforming general neural operators, e.g., FNO and transformer-based neural operators (GNOT, Transolver).\n\nThe paper clearly presents the method, and comprehensive experiments are well deployed to show its advantage. DNS is a challenging problem of high significance, and the method is showing strong evidence (in terms of accuracy and speed) of practical applications of neural operators in CFD.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets deep learning in the context of turbulence simulations, motivating its approach with the well established self similarity of eddies in the turbulence cascade. An architecture is proposed that uses spectral elements in the context of Transformer models, processing inputs in terms of s subgrid-scale and and LES branch. This idea is a classic one, and well-established in turbulence learning [1]. Unfortunately, the paper does not acknowledge this, and instead tries to present this idea as a new contribution, without citing the classic method. While there are clearly differences to [1], the job of the authors would be to discuss and evaluate the differences clearly. Unfortunately, the lack of context and lack of previous work comparisons is continuing later on in the paper.\n\nIn the proposed architecture, the SGS part uses FNOs, which are combined with attention based methods for the second branch. The FNO part swaps the FFT for SEM basis functions. The advantages of the SEM approach over an FFT are claimed as a contribution, but its advantages are not made clear in a direct comparison. The FNO approach also means it inherits the negative aspects: especially the bad scaling of FNOs for 3D seems problematic here. Global basis functions for larger 3D resolutions are inherently problematic, which is, however, mitigated with the axial attention.\n\nScale separation is done via a spectral cutoff, the method uses a manual split into low and high frequencies in each. Low frequencies are provided to the high-frequency branch - in L140 the authors discuss this as surprising, but it's directly in line with the primary scattering of the turbulence cascade. That the other direction is not helpful points to limited accuracy: it's the classic \"closure problem\" of turbulence, and in the general case the small scales should play a role once the overall approach is sufficiently accurate. (I would recommend to change this paragraph accordingly.)\n\nDue to the lower dimensionality of the low \"LES\" frequencies, a Transformer with dense attention is applied. The tokens for this stage are likewise processed via the SEM basis functions.\n\nThe paper evaluates the method with a learned predictions for 3D isotropic turbulence, and a solver-in-the-loop case for 2D kolmogorov flows. Here, likewise, the authors only focus on the work by Kochkov et al., but do not provide context of their method for newer works from the turbulence area, e.g. [2,3] just to name a few. Generally, the writing and exposition of the paper is problematic - the paper tries to focus on turbulence in its outset, but then seems to turn into a paper focusing on NN architectures. \n\nThe authors have also moved related work and the discussion of limitations to the appendix. This naturally frees up room in the main paper, but the NeurIPS page limit is there for a reason. Given the requirements for a NeurIPS paper, these parts should be included in the main text. And naturally, for a paper targeting turbulence, a discussion of ML approaches in this area would be necessary in the main text (not only citing classic text books).\n\nThe different methods focus on time stepping, but the paper is unfortunately unclear on the details here. The authors provide details on unrolling (like \"N=5\"), but don't specify for how many steps the networks were evaluated. Some tests seem to use integer time steps, others like in table 8 use smaller ones. This is crucial information to evaluate the time stability of the method.\n\n[1] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. \nTowards physics-informed deep learning for turbulent flow prediction. SIGKDD, 2020\n[2] Bjoern List, Liwei Chen, and Nils Thuerey.\nLearned Turbulence Modelling with Differentiable Fluid Solvers. JFM, 2022\n[3] V. Shankar, D. Chakraborty, V. Viswanathan, and R. Maulik.\nDifferentiable turbulence: Closure as a partial differential equation constrained optimization. PRF, 2025\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- 3D PDEs , and turbulence are tough and interesting tasks. They're definitely worth studying, and not many papers address 3D up to now.\n\n- the paper shows some strong results: the isotropic turbulence case with 256^3 is neat to see. This is a high resolution, and the \"fair\" comparison with a high-resolved baseline makes this a strong case\n\n- the paper comes with a good set of \"generic\" baselines that, however, seem quite arbitrary: the data is structured, then why compare to \"unstructured\" methods like GNOT and Transolver? \n\n- it's good to see the architecture behaves like a convolutional network and can deal with different domain sizes\n\nWeaknesses\n\n- despite focusing on turbulence, the paper does not provide context of previous work for turbulence-focused ML approaches \n\n- the central SGS/LES split was proposed in previous work many years ago [1]; the authors do not seem to be aware of this; a direct comparison with tf-Net is missing\n\n- the contribution of an SEM approach is likewise claimed as contribution, but it is not mdke clear how much improvements SEM provides over an FFT basis (the Legendre vs Chebyshev comparison implies that the basis does not have a big influence)\n\n- the details on the time discretizations are missing. without this information, the results are difficult to interpret\n\n- similarly, details of the spectral cutoff for the SGS vs LES split are missing\n\nSo overall, I think the paper makes some promising steps, but my impression is that it needs a larger, properly checked revision before publication can be recommended.\n\nQUESTIONS:\n- can the authors demonstrate that the SEM approach has benefits over an FFT by swapping out the basis functions within the full proposed architecture?\n\n- what are the time steps, and how many rollout steps evaluated? \n\n- how is the spectral cutoff chosen, and how much influence does it have. I.e., what is k_max in each case, and how much influence does it have in practice? An ablation would be important here to whether the scale separation actually has a positive influence.\n\nLIMITATIONS:\nLimitations are not discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EddyFormer for ML-based modeling of turbulence. The main idea is to take inspiration from large-eddy simulations and use two sub-networks: standard-attention for modeling large-scale and spectral convolutions for modeling subgrid-scale turbulence. EddyFormer is benchmarked on different turbulence problems from the Well dataset. Results show that EddyFormer outperforms neural operator (FNO, FFNO, TFNO variants) and transformer (GNOT, Transolver) baselines. The paper also reports 30x speedup over DNS simulations using efficient pseudo-spectrak solvers on a single GPU.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n-- Builds on turbulence theory and well-established modeling methodologies (i.e., LES) motivating the dual-stream design underlying EddyFormer. \n\n-- Experimental evaluations consider challenging benchmarks such as Rayleigh-Taylor instability and demonstrates improved performance compared to baselines.\n\nWeaknesses: \n\n-- The evaluation of long-time rollout is limited in duration. If delta t = 1 s, the maximum number of time steps in the examples is atmost 50. \n\n-- The design lacks any specific treatment for flows with wall effects and is unclear how the current architecture will generalize to real-world turbulence with irregular geometries or wall-bounded flows. \n\n-- Evaluation metrics are limited to relative L2 and doesn't check if physical quantities are conserved. Since the model also doesn't enforce any hard physical constraints (e.g., conservation laws), it's unclear if conservation is met and how these models will generalize to OOD.\n\n-- The reported speedup is for a specific problem size (256^3) for a specific benchmark. It is unclear how EddyFormer scales at larger sizes and how that compares to DNS scaling. Quadratic scaling in attention costs might limit the scalability of the model at higher resolutions.\n\nQUESTIONS:\nPlease see weaknesses listed above.\n\nLIMITATIONS:\nData efficiency is not addressed. There is an underlying assumption that there is access to large training data for transformer models but this is not the case in turbulence (specifically DNS data as used in this paper).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-20"}