SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To assemble the Jacobian matrix  $J(\theta(t))$ , the gradient of  $\hat{u}$  is evaluated on samples generated from the spatial domain. If not noted otherwise, we use 10 000 equidistant points for Allen-Cahn and Burgers' and 200 000 equidistant points for Vlasov. For the time-optimized RSNG results in Table 1 (rows 4 and 8) we use 1000 equidistant points for Allen-Cahn and Burgers'. In the dense and sparse least-squares system we regularize the direct solver so as to avoid numerical instability. For this we set the rcond parameters in numpy implementation of 1st sq. The values used are 1e-4, 1e-4, 1e-5 for Allen-Cahn, Burgers', and Vlasov respectively.
A.6 Global Methods Benchmark
Here we detail the training setups for our benchmarks of the global methods given in Table 1.
For all PINN experiments we sampled data on a grid with 100 points in the time domain and 256 points in the spatial domain. All PINNs were trained with the following hyperparameters:
- optimizer : Adam then L-BFGS
- spatial samples : 256
time samples:100
- activation : tanh
For the plain PINN experiments our architecture is a MLP with layers sizes as follows: [2, 128, 128, 128, 128, 1]. Boundary and initial conditions were enforced through a penalty in the loss function.
For the Causal PINNs we use the architecture described in the original paper, with periodic embedding and "modified MLP layers." The layer sizes were as follows: [periodic embedding, 128, 128, 128, 128, 1]. Our implementation uses much of the original code provided in [51]. We used only one time-window for training as this is what was chosen in [51] for the Allen-Cahn equation. The tolerance hyperparameter, which controls the degree to which the loss function enforces causal training, was set to 100 and 50 for the Allen-Cahn and Burgers' equations respectively. Additionally the  $\lambda_{ic}$  parameter, which controls the loss functions' weighting on the initial condition, was set to 100 and 1 for the Allen-Cahn and Burgers' equations respectively. These were both chosen via a hyperparameter search. For details on the context and meaning of these hyperparameters see the original paper [51].
The PINNs trained for Allen-Cahn used 1000 steps of Adam followed by 30000 steps of L-BFGS. For Burgers' equations we used 1000 steps of Adam followed by 60000 steps of L-BFGS. More steps are needed for Burgers' equation in order to sufficiently resolve the sharp gradient in the solution due to the low viscosity number. We similarly choose a smaller timestep for Burgers' in the Neural Galerkin schemes. We note that fewer optimization iterations can be used for training PINNs but this resulted in much larger errors in our experiments. In any case, widely varying the number optimization iterations and other PINNs hyperparameters did not achieve errors in the range of what RSNG achieves in these examples.