Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent PDEs

Authors: jan  hagnberger
marimuthu  kalimuthu
mathias  niepert
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-10 14:48:43

Abstract: Transformer models are increasingly used for solving Partial Differential Equations (PDEs). However, they lack at least one of several desirable properties of an ideal surrogate model such as (i) generalization to PDE parameters not seen during training, (ii) spatial and temporal zero-shot super-resolution, (iii) continuous temporal extrapolation, (iv) applicability to PDEs of different dimensionalities, and (v) efficient inference for longer temporal rollouts. To address these limitations, we propose Vectorized Conditional Neural Fields (VCNeFs) which represent the solution of time-dependent PDEs as neural fields. Contrary to prior methods, VCNeFs compute, for a set of multiple spatio-temporal query points, their solutions in parallel while also modeling their dependencies through attention mechanisms. Moreover, VCNeF can condition the neural field on both the initial conditions and the parameters of the PDEs. An extensive set of experiments demonstrates that VCNeFs are competitive with and often outperform existing ML-based surrogate models.

Main paper: 
 introduction: 
 The simulation of physical systems often involves solving PDEs and Machine Learning (ML) based surrogate models are increasingly used to address this challenging task (Lu et al., 2019; Li et al., 2020b; Cao, 2021) . Utilizing ML methods for solving PDEs has several advantages such as faster simulation time than classical numerical PDE solvers, differentiability of the surrogate models (Takamoto et al., 2022) , and their ability to be used even when the underlying PDEs are not known exactly (Li et al., 2020a) .Transformers (Vaswani et al., 2017) and its variants are successfully used in natural language processing (Devlin et al., 2018) , speech processing (Gulati et al., 2020) , and computer vision (Dosovitskiy et al., 2020) . Additionally, the use of Transformer models in Scientific Machine Learning (SciML) to model physical systems (Geneva & Zabaras, 2020) and solve PDEs (Cao, 2021; Li et al., 2023a; b; McCabe et al., 2023) is steadily increasing. Meanwhile, recent advances in neural networks for computer graphics tasks have introduced Neural Fields (Xie et al., 2021) , which have proven to be an efficient method to solve PDEs (Sitzmann et al., 2020; Chen et al., 2023b; a; Yin et al., 2023; Serrano et al., 2023) . Despite these recent advances in neural architectures for PDE solving, current methods lack several of the characteristics of an ideal PDE solver: (i) generalization to different Initial Conditions (ICs), (ii) PDE parameters, (iii) applicability to PDEs of different dimensionalities, (iv) stability over long rollouts, (v) temporal extrapolation, (vi) spatial and temporal super-resolution, all with affordable cost, high speed, and accuracy.Towards developing a model that encompasses these ideal characteristics, we propose Vectorized Conditional Neural Field (VCNeF), a transformer-based conditional neural field, that solves PDEs continuously in time endowing the model with temporal as well as spatial Zero-Shot Super-Resolution (ZSSR) capabilities. The model introduces a new mechanism to condition a neural field on ICs and PDE parameters to achieve generalization to both ICs and PDE parameter values not seen during training. While modeling the solution using neural fields such as Physics-Informed Neural Networks (PINNs) naturally provides temporal and spatial ZSSR, these methods are inefficient since we need to query them separately for every spatial and temporal location. We achieve faster training and inference by vectorizing the computations on GPUs. Moreover, the proposed method also explicitly models dependencies between model queries for different spatial and temporal coordinates. 
method: 
 In this section, we briefly recall (conditional) neural fields and relate them to solving parametric PDEs. Furthermore, we propose Vectorized Conditional Neural Fields.Neural Fields and Conditional Neural Fields. Neural Fields (NeFs; Xie et al. (2021) ) learn a function f which maps the spatial and temporal coordinates (i.e., x ∈ R D , t ∈ R + respectively) to a quantity q ∈ R c . More formally, a neural field can be expressed as a function f θ : (R + × R D ) → R c with (t, x) → q (1) that is parametrized by a neural network with parameters θ. For solving PDEs, the function f θ models the solution function u and the quantity q represents the solution's value for the different channels, each representing a physical quantity (e.g., density). PINNs (Raissi et al., 2017) are a special case of neural fields with a physics-aware loss function, modeling the solution u asEQUATION) where f θ denotes the neural field that maps the input spatial and temporal locations to the solution of the PDE. Conditional Neural Fields (CNeFs; Xie et al. (2021) ) extend NeFs with a conditioning factor z to influence the output of the neural field. The conditioning factor was originally introduced for computer vision to control the colours or shapes of objects. In contrast, we condition the neural field, which models the solution of the PDE, on the IC and the PDE parameters (cf. Figure 1 ). Thus, the conditioning factor influences the entire field variable.Vectorized Conditional Neural Fields. Typically, a (conditional) neural field generates the output quantities for all input spatial and temporal coordinates in multiple and independent forward passes.The training and inference times can be improved by processing multiple inputs in parallel on the GPU which is possible since all forward passes are independent and, hence, embarrassingly parallel. However, there are spatial dependencies between different input spatial coordinates, particularly for solving PDEs, that will not be exploited with CNeFs or by processing multiple inputs of CNeFs in parallel. Consequently, we propose extending CNeFs to take a vector with arbitrary spatial coordinates of variable size as input, exploit the dependencies among the input coordinates when generating the outputs, and regress all outputs for the inputs in one forward pass. Hence, we name our proposed model Vectorized Conditional Neural Field since it implicitly generates a vectorization of the input spatial coordinates for a given time t. In formal terms, VCNeF represents a function where u(t, x i ) denotes the PDE solution for the spatial coordinates x i . Note that we do not impose a structure on the spatial coordinates x i and that the number of spatial points (i.e., s) can be arbitrary.EQUATIONThe model can process multiple timesteps t in parallel on the GPU to further improve the training and inference times since VCNeF does not exploit the dependencies between the temporal coordinates. EQUATIONwhere θ represents the parameters of the neural network, X ∈ R s×D the grid of dimensionality D with query spatial coordinates, t the query time, u(0, X) the IC, and p the PDE parameter(s). u(t, X; u(0, X), p) denotes the solution function, which depends on the given IC and PDE parameter(s), that is directly regressed by VCNeF. R s×c stands for the entire solution field with c channels. in parallel since the predictions of u(t, •) are independent of each other and, hence, embarrassingly parallel. 
vcnef features: 
 Physics-Informed VCNeF. The loss function of VCNeF can be easily extended with a physicsinformed loss as in PINNs (Raissi et al., 2017) since VCNeF directly models the solution function u and, therefore, the derivatives can be computed with automatic differentiation (Maclaurin et al., 2015; Paszke et al., 2017) . 
experiments: 
 We design our experiments such that we are able to answer the following research questions: We conduct experiments on the following hydrodynamical equations of parametric PDEs from PDEBench (Takamoto et al., 2022) : 1D Burgers', 1D Advection, and 1D, 2D, and 3D Compressible Navier-Stokes (CNS) equations. Unlike the prevalent models in the SciML literature, we train and test the models with a single timestep as IC and predicting multiple future steps, as this setting is best suited for real-world applications. We choose FNO (Li et al., 2020a) , MP-PDE (Brandstetter et al., 2022) , CORAL (Serrano et al., 2023) , Galerkin Transformer (Cao, 2021) , and OFormer (Li et al., 2023a) as the SOTA baselines. The predictions of FNO, Galerkin Transformer, and OFormer are achieved in an autoregressive fashion, while VCNeF predicts the entire trajectory of the simulation directly in one forward pass.RQ1. We test VCNeF's generalization ability to different ICs by evaluating the models on the corresponding test sets of PDEBench. test them on an unseen set of parameter values. Figure 2 shows the error distribution for 1D CNS. We observe that VCNeF generalizes better to unseen PDE parameters than the compared baseline models.RQ3. We evaluate the ability of VCNeF achieving spatial and temporal ZSSR which means that the model is trained on a reduced spatial and temporal resolution and then used for high-resolution inference. The results in Table 16 show that OFormer and our proposed VCNeF have spatial ZSSR capabilities since there is no significant increase in error even when the spatial resolution is 4x the training resolution. For the case of FNO on 1D Burgers, we observe a near multiplicative increase of error as the factor of resolution increases (2x and 4x). The inherent support of continuous-time inference of VCNeF allows predicting a trajectory with a smaller temporal step size than encountered during training. Table 17 shows a negligible increase in error for VCNeF exemplifying the model's superior temporal ZSSR capabilities by learning the dependency between the solution and time. Doing interpolation between timesteps seems to be only effective for smooth targets such as CNS.RQ4. Lastly, we measure the inference times of the proposed and baseline models. The times are the raw inference times, without measuring the time needed to transfer the data from the host device to the GPU, on a single NVIDIA A100-SXM4 80GB GPU. We measure the time of the models predicting 40 to 240 timesteps in the future with a spatial resolution of s = 256. The times in Figure 7 demonstrate that VCNeF is significantly faster than the other transformer-based counterparts. However, the speed-up results in a higher GPU memory consumption as shown in Table 18 . 
conclusion: 
 In this work, we have designed an effective Neural PDE Solver, Vectorized Conditional Neural Field, based on the conditional neural fields framework, and demonstrated its generalization capabilities across multiple axes of desiderata: spatial, temporal, ICs, and PDE parameter values. Furthermore, the proposed model allows to trade-off the inference time or GPU memory when generating the solutions. As a future work, we aim to experiment on turbulent flow simulations and incompressible Navier-Stokes equations, improve the VCNeF model design further with adaptive timestepping, investigate sophisticated strategies for conditioning the neural fields (Rebain et al., 2023) , test physicsinformed losses (Li et al., 2021b) , and analyze the model scaling behaviour on vast amounts of PDE data, including multiphysics simulations (McCabe et al., 2023; Sun et al., 2024) . 
 
 Appendix: A Related Work A.2 TRANSFORMERS FOR SOLVING PDES Transformers are increasingly being utilized for modelling physical systems or PDEs. Previous works can be divided into using Transformers for applying temporal self-attention (Geneva & Zabaras, 2020) to model the temporal dependencies or applying spatial self-attention for capturing spatial dependencies of the PDE (Cao, 2021; Li et al., 2023a; b) . Applying the spatial self-attention as in Fourier and Galerkin Transformer (Cao, 2021) , or in OFormer (Li et al., 2023a ) yields a neural operator (Kovachki et al., 2021) endowing the model with spatial ZSSR capabilities. Since these models do not consider time as an additional input, they are not time-continuous and, hence, fixed to a trained temporal discretization. To remedy the issue, the diffusion-inspired temporal Transformer operator (Ovadia et al., 2023) uses the time component to condition the input solution, thereby supporting a flexible temporal discretization for inference.Although ML-based methods have shown great success in solving PDEs, they often do not consider PDE parameters as input resulting in failures to generalize to unseen parameter values. Recent works such as CAPE (Takamoto et al., 2023) , MP-PDE (Brandstetter et al., 2022) , and PDERefiner (Lippe et al., 2023) consider the PDE parameters as additional model input. Along the same lines, our proposed model also considers the PDE parameter to improve the generalization error of unseen PDE parameter values.Neural Fields (NeFs) has become widely popular in signal processing (Sitzmann et al., 2020) , computer vision (Mescheder et al., 2019) , computer graphics (Chu et al., 2022) , and recently in SciML for solving PDEs (Chen et al., 2023b; Yin et al., 2023; Chen et al., 2023a; Serrano et al., 2023) .The prevalent INR models for PDE solving follow the "Encode-Process-Decode" paradigm. DiNO (Yin et al., 2023) has an encoder, a Neural ODE (NODE) to model dynamics, and a decoder. CORAL, an improvement to DiNO, has a two-step training procedure whereby the input and output INR modules with shared parameters are trained first, and subsequently, the dynamics modeling block is trained using the learned latent codes. DiNO and CORAL utilize an INR to encode and decode the PDE solution in a latent space and a NODE to propagate the dynamics in latent space, while our approach utilizes a neural field to represent the entire PDE solution encompassing both the spatial and temporal dependencies within a shared space.A.5 COMPARISON OF NEURAL ARCHITECTURES FOR PDE SOLVING ✓ ✗ ✓ ✗ ✗ OFormer ✓ ✗ ✓ ✗ ✓ cFNO ✓ ✓ ✓ ✗ ✗ cOFormer ✓ ✓ ✓ ✗ ✓ Neural Field PINN ✗ ✗ ✓ ✓ ✗ Conditional Neural Field CORAL ✓ ✗ ✓ ✓ ✗ VCNeF (ours) ✓ ✓ ✓ ✓ ✓Table 2 : Overview of distinct properties of benchmark baselines and our proposed VCNeF model.We propose a transformer-based VCNeF that applies self-attention to the spatial domain to capture dependencies between the spatial coordinates. The input spatio-temporal coordinates and the physical representation of ICs are represented in a latent space. Both latent representations are fed into modulation blocks that capture spatial dependencies and condition the coordinates on the IC. The output of the modulation blocks, which represent the solution, is then decoded to obtain the representation in the physical space.Latent Representation of Coordinates. The input coordinates, consisting of the query time t ∈ R + that determines the time for which the model's prediction is sought and the spatial coordinates to encode the spatial coordinates are used.x i ∈ R D ,For 1D:EQUATIONwhere ∥ denotes the concatenation of two vectors.Latent Representation of IC. The input IC is mapped to a latent representation by either applying a shared linear layer to each solution point u(t, x i ) or by dividing the spatial domain into nonoverlapping patches and applying a linear layer to the patches, akin to Vision Transformers (ViTs; Dosovitskiy et al. (2020) ). However, unlike a traditional ViT, our patch generation has two branches: patches of a smaller size (4 × 4) and of a larger size (16 × 16) as proposed in Chen et al. (2021) since we aim to capture the dynamics accurately with multi-scale processing.For 1D: zEQUATIONA vector in the latent space (i.e., token) either represents the solution on a spatial point (for 1D) or the solution on a patch of spatial points (for 2D and 3D). The grid contains the coordinates where the solutions are sampled in the spatial domain. This information is used when generating the latent representations of the IC to ensure that each latent representation has information about the position. The PDE parameters p are also added to the latent representation. We neglect additional positional encoding to prevent any length generalization problems (Ruoss et al., 2023) that could prevent changing the spatial resolution after training.Linear Transformer Encoder for IC. We utilize a Linear Transformer (Katharopoulos et al., 2020) with self-attention in our VCNeF architecture to generate an attention-refined latent representation of the IC Z (0) . The global receptive field of Transformer allows the proposed architecture to capture spatial dependencies in the IC, although each token contains only partial spatial information. Intuitively, the Transformer outputs latent representations that incorporate the entire spatial solution and not only a single spatial point or a subset of spatial points. We assume that this is beneficial to generate a better representation of the IC to condition the input spatio-temporal coordinates accordingly.Z (n+1) = Transformer Block Z (n) (7)where Transformer Block(•) is a Linear Transformer block with self-attention and n denotes the n th block.Modulation of Coordinates based on IC. The modulation blocks condition the input coordinates on the input IC Z (3) by modulating the latent representation C of the coordinates. The block contains self-attention, a non-linearity σ, a modulation mechanism similar to Feature-wise Linear Modulation (FiLM; Perez et al. ( 2018)), layer normalization, residual connections, and an MLP. However, the conditioning mechanism uses only the scaling (i.e. pointwise multiplication) of FiLM and omits the shift (i.e. pointwise addition). A modulation block is expressed succinctly asZ (m+1) = Modulation Block C, Z (m) = MLP σ Self Attn Z (m) • MLP(C) σ(X) = ELU(X) + 1 (8)where Z (3) ∈ R s×d represents the IC, C ∈ R s×d denotes the latent representation of the input coordinates, • represents the Hadamard product, and m is the m th fusion block. The residual connections and layer normalization are omitted in Equation 8for the sake of simplicity.Decoding the Solution's Latent Representation. The solution's latent representation Z ( 6) is mapped back to the physical space by either applying an MLP for 1D or by mapping the latent representations to small and large patches and outputting the weighted sum of the small and large patches for 2D and 3D.We conduct an ablation study on the prominent parts of the proposed architecture. Namely, the self-attention mechanism that allows the model to capture spatial dependencies and the conditioning mechanism that is used to condition the neural field. For 2D PDEs, we also study the effect on the model's performance for patches of one size and the multi-scale patching mechanism with small and large patches. Additionally, we compare linear attention (Katharopoulos et al., 2020) with vanilla attention (Vaswani et al., 2017) in terms of training time and GPU memory consumption.For simplicity, we perform the ablation study mainly on the 1D Burgers' and 2D CNS datasets.Self-Attention, Conditioning Mechanism, and Patch Generation. Table 3 shows the results of the proposed model with and without self-attention as well as with different modulation mechanisms to condition the neural field. This part serves as an expository section on the parametric PDE datasets of the hydrodynamical equations that we experimented with to answer our research questions.The Burgers' PDE models the non-linear behaviour and diffusion process in fluid dynamics and is expressed asEQUATIONwhere the PDE parameter ν denotes the diffusion coefficient. Our dataset contains solutions for x ∈ (-1, 1) with a maximum resolution of 1024 spatial discretization points and t ∈ (0, 2] with a maximum resolution of 201 temporal discretization steps including the initial condition. We subsample the data along the temporal and spatial domain yielding a trajectory of 41 time steps where each snapshot has a spatial resolution of 256, respectively.The Advection PDE models pure advection behaviour without non-linearity. It is written asEQUATIONwhere the PDE parameter β denotes the advection velocity. Similar to 1D Burgers', we subsample the data to get samples with a trajectory of 41 time steps each with a spatial resolution of 256.Following Takamoto et al. (2022; 2023) , we conduct experiments on the challenging Compressible Navier-Stokes datasets, with random initial fields as the initial conditions for 1D, 2D, and 3D cases. EQUATIONEQUATION∂ t (ϵ + ρv 2 2 ) + ∇ • [(p + ϵ + ρv 2 2 )v -v • σ ′ ] = 0,where ρ represents the (mass) density of the fluid, v the fluid velocity (in vector field), p the gas pressure, ϵ the internal energy according to the equation of state, σ ′ the viscous stress tensor, η and ζ the PDE parameters which represent the shear and bulk viscosities, respectively. We subsample the data for 1D, 2D, and 3D equations. The original resolution of 1D simulation has 1024 spatial points and 101 timesteps for each trajectory. For training purposes, we subsample across both spatial and temporal resolutions by a factor of 4 and 2 respectively yielding a trajectory of length 51 time steps and a spatial resolution of 256. To be consistent with other 1D PDE trajectory lengths, we retain only the first 41 timesteps and perform experiments on this temporally truncated data. The original resolution of the 2D simulation data is 128 × 128 for each channel (i.e., density, velocity-x, velocity-y, and pressure) and has 21 timesteps. For training, we have subsampled the data only for the spatial dimensions resulting in a resolution of 64 × 64. For 3D data, the original spatial resolution is 128 × 128 × 128 which was subsampled to a resolution of 32 × 32 × 32 for training.Fourier Neural Operator (FNO; Li et al. (2020a) ) is an implementation of a neural operator that maps from one function a(x) to another function a ′ (x) (Kovachki et al., 2021) . Traditionally, neural networks learn a mapping between two finite-dimensional Euclidean spaces which leads to the problem that they are fixed to a spatial resolution when used for solving PDEs. Neural Operators overcome this limitation by learning an operator that is a mapping between infinite-dimensional function spaces (i.e., mapping between functions). The function a(x) := u(t n , x) represents the solution for timestep t n and a ′ (x) := u(t n+1 , x) the solution for a future timestep t n+1 . FNO, an instantiation of a neural operator, is based on spectral convolution layers which implement an integral transformation of the input function. The integral transformation is implemented with discrete Fourier transforms on the spatial domain allowing an efficient and expressive architecture. We use the FNO model as a baseline because it is a very strong model in terms of accuracy, speed, and memory consumption. The FNO model is trained in an autoregressive fashion for 500 epochs.D.1.1 CFNO cFNO (Takamoto et al., 2023) is the adapted version of the FNO where the PDE parameters are added as an additional channel to condition the model on the PDE parameters.Several models exist that use Graph Neural Networks (GNNs) for solving PDEs (Brandstetter et al., 2022; Boussif et al., 2022) Message Passing Neural PDE Solvers. MP-PDE (Brandstetter et al., 2022) follows the prevalent Encode-Process-Decode framework for simulating physical systems (Sanchez-Gonzalez et al., 2020) . The MP-PDE model has an MLP as encoder, GNN as a processor, and a CNN as the decoder.Moreover, the model introduces several tricks such as pushforward, temporal bundling (time window with 5 timesteps), and random timesteps in the length of the trajectory as starting points during training for autoregressive PDE solving, while also considering the PDE parameter values as additional input, making it as a versatile choice for generalized neural PDE solving. Hence, we adopt it as a baseline. However, it has to be noted that we apply the model only to 1D PDEs. The configuration of our adaptation for 1D PDEs amounts to 614,929 model parameters which is comparable to the other baselines.Considering that CORAL (Serrano et al., 2023) , to the best of our knowledge, is the current state-ofthe-art INR-based method for solving PDEs, we benchmark our proposed VCNeF model against it on 1D Advection and Burgers' as well as on the challenging 1D and 2D compressible Navier-Stokes PDEs (Takamoto et al., 2022) . The CORAL model is trained purely in a data-driven manner and involves two stages: In traditional numerical solvers, the simulation time of trajectories of a given PDE is influenced by several factors such as the value of PDE parameter, efficiency of software implementation, the type and order of numerical algorithm, fineness of discretization meshes, length of the domain, and so on. On the contrary, inference (simulation) time of ML models is agnostic to these factors, which is one of the huge advantages of machine-learned PDE surrogates.To compare and understand the time consumed for longer rollouts, we visualize the inference times of VCNeF, FNO, Galkerin Transformer, and OFormer models. Figure 7 shows the scaling behaviour of the inference times for a variable number of timesteps in the future {80, 120, 160, 200, 240}. Table 18 includes the memory consumption in addition to the inference times. The results demonstrate that the inference times of the proposed VCNeF model scale better when compared to other transformer-based baselines and are competitive with FNO. However, the speed-up results in a higher memory consumption. The model can also be used to do inference in a sequential fashion which reduces the memory consumption but increases the inference time. Nevertheless, it is still faster than OFormer while the memory requirement remains the same even for extended rollout durations making it as a strong contender among the transformer-based neural PDE solvers. We encode the parameter values as an additional channel. 


Decision: The reviewers mention that this is an interesting approach, where the vectorization of neural fields is novel. I agree with Reviewer tiYc that the experimental results are limited and it is hard to draw conclusions on the effectiveness of the method but I feel that for preliminary work for the workshop it is acceptable. I also agree that the terminology Neural Fields is unnecessary. I vote for acceptance pending that the authors address the comments by the reviewers and add the references from Reviewer NHSb in the camera-ready version.

Reviews: review: title: title: decent paper with a meaningful solution, but misses the literature and fair comparison with state of the art methods
main_review: main_review: This submission deals with fixing the caveats of transformer networks for solving PDEs. Some of the caveats include generalization to PDE parameters not seen during training,(ii) spatial and temporal zero-shot super-resolution,(iii) continuous temporal extrapolation, (iv) dimensionality generalization of PDEs, and (v) efficient inference for longer temporal rollouts. To solve these issues this work utilizes the attention mechanism to design conditional vectorized neural fields (VCNeF) that can achieve fast training and inference by parallelization on GPUs. The performance is compared with FNO and OFormer and it shows some gains. 

This submission addresses an important and timely problem. The proposed solution is also meaningful. The comparison with FNO doesn't seem fair. There are new versions of FNO such as AFNO [1] based on transformer networks and attention that is scalable and fast to train. Also, SFNO [2] seems to improve the role out. I would recommend the authors to compare these two works and compare in the experiments  if possible.

[1] Guibas J, Mardani M, Li Z, Tao A, Anandkumar A, Catanzaro B. Efficient token mixing for transformers via adaptive fourier neural operators. InInternational Conference on Learning Representations 2021 Oct 6.

[2] Bonev, Boris, et al. "Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere." arXiv preprint arXiv:2306.03838 (2023).
title: title: Potentially interesting idea with little to back it up experimentally
main_review: main_review: Verdict:
An interesting proposal with very little to back up the effectiveness of the method. The main novelties of this work seems to be: 1) the treatment of inputs and outputs as "neural fields" or in other words, inputs and outputs parameterized as functions. 2) vecrtorization of this neural fields 3) usage of transformers with these NN-valued inputs/outputs While 1 and 3 are not novel in itself see e.g. [1,2], I have not seen 2) yet. Where the paper breaks apart in my view is the sufficient evaluation and scientific documentation. The examples on which the architecture is evaluated are mere 1d toy examples. Moreover, parameter counts and hyperparameters are not disclosed, which makes it difficult to evaluate the effectiveness of the method is in comparison to the baseline models. Moreover, it is unclear to me what the trade-offs of this method are from the discussion presented in the paper.

For the above reasons, I am inclined to reject the paper.

Minor comments:
- I find the terminology "Neural fields" somewhat unnecessary in the realm of SciML. These are simply PDE solutions represented by Neural Networks. It's unfortunate that ML in general tends to favour trendy names such as Neural Fields, and would recommend sticking to clear-cut terminology wherever possible
- I would name it Appendix rather than tailpiece

References
[1] Yifan Du, Tamer A. Zaki - Evolutional Deep Neural Network (https://arxiv.org/abs/2103.09959)

[2] Louis Serrano, Lise Le Boudec, Armand Kassaï Koupaï, Thomas X Wang, Yuan Yin, Jean-Noël Vittaut, Patrick Gallinari - Operator Learning with Neural Fields: Tackling PDEs on General Geometries (https://arxiv.org/abs/2306.07266)