Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper builds on recent advances in deep learning for physical simulation to reduce the computational complexity of learned surrogate models by avoiding local evolution. Instead of updating the values at each discretized point in space, the authors propose to evolve the model in a global latent space. By encoding both states and boundaries into lower-dimensional latent vectors, the model can evolve dynamics in the latent space and recover the inferred states only when needed. The paper also introduces techniques for training, such as a new consistency loss directly in latent space, a fast way of dealing with inverse optimization problems by backpropagating through time in the latent space, and an annealing technique for boundary optimization. Experiments in various PDE settings demonstrate that the proposed model, LE-PDE (Latent-Evolution of PDEs) compares competitively against state-of-the-art methods while requiring a fraction of the other methods’ computational resources.
strengths_and_weaknesses: ### Strenghts

The paper is appropriately placed in the current growing literature on scientific machine learning to create a novel fast model for tackling complex problems in the realm of PDEs. The exposition for the motivation is written crisply and is generally easy to follow. The experiments are performed in challenging settings, such as boundary control with the 2D Navier Stokes equations. The results show that very large neural networks are not always necessary to represent complex physics and that “simple” latent MLPs with few parameters can learn such physics evolutions thus considerably lowering computational requirements, which is significant in deep learning for simulation.

### Weaknesses

Although the speedups of LE-PDE are considerable, it is unclear how the proposed approach would compare against other models with a similar number of parameters. What if the FNO had fewer parameters, comparable to LE-PDE? Would it still be better than the proposed approach? In other words, it is unclear whether the method is *Pareto efficient* or not since there is no Pareto plot considering an extensive hyperparameter search for both LE-PDE and close competitors such as the FNO.

About the novelty: there is a possibly missing reference [1] (although not published in conference proceedings), in which the authors evolve a model in the latent space (see Figure 1 and Section 3.1 of the reference), and the main idea of the paper is very similar to the presented one. How does this work compare against it?

The experimental section in the Appendix about the 3D extension of the method (3D Navier Stokes) lacks some experimental details and evaluation. The learned model is not compared with other deep learning approaches, so it is difficult to assess how the proposed method would perform against other models (except for the same version of the model without the global latent space evolution). While the comparison in the main text is against other deep learning approaches, here it is against the ground truth solver which is known to be generally much slower than deep learning approaches. Moreover, there is no report on statistical errors but only a qualitative plot (that also shows noticeable artifacts).

[1] Sanchez A, Kochkov D, Smith JA, Brenner M, Battaglia P, Pfaff TJ. Learning latent field dynamics of PDEs,  Third Workshop on Machine Learning and the Physical Sciences (NeurIPS 2020)

### Minor comments

These are primarily typos and do not influence my score:

- Line 235-236: “Burgur’s Equation”
- Line 355: “sweep spot”
- Line 707 (Appendix): "The details of the dataset has already given in […]”
- Line 748: “To explore how LE-PDE to larger scale turbulent dynamics a […]”
questions: 1. GNN+MLPs as a future direction, but it is unclear to me whether this could be possible. How would message passing be realized in a global latent space, though? Could we still use the same latent space even if we added particles or mesh points?

2. About the boundary annealing, what is meant by $\beta$ being a “temperature hyperparameter”? This notation is also not introduced in the main text.
limitations: See above. As stated in the main text, no major negative societal impact is to be expected from this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a method LE-PDE to efficiently perform forward simulation and inverse optimization of Partial differential equation based models by performing the evolution in latent space. A loss term penalizing 3 terms including the consistency between latent space and the original space is defined, and optimized via backpropagation for the inverse task. Experimental evaluations including an ablation study is provided.

strengths_and_weaknesses: Strength:
The authors put together a very clear introduction and motivation for their work, the method is described clearly in an easily reproducible manner.

Weakness:
The authors state that low dimensional representation exists for high dimensional data (line 49-64). Please have a look at Johnson-Lindenstrauss lemma and cite appropriate literature applicable here.

The authors may want to try their experiments on a larger scale. Grid size of 64 is too small, as the authors mention dimensions in the millions/billions in the introduction, at least one experiment should demonstrate the efficacy of the proposed method in such a larger scale where other baseline methods are computationally very very expensive and resource consuming.

Another concern about the proposed method is the applicability of the proposed system in real-world scenarios. Experimental data are often noisy. The authors may want to look at how noise affects the latent space evolution and encoder-decoder performance.

questions: Please have a look at the weakness mentioned in the strength and weakness section and address these. Overall the idea seems interesting, the authors need to substantiate their claims in light of existing literature and possibly a few more experiments.
limitations: Yes, the limitations are discussed in the appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper looks at the problem of expensive time cost when simulating the time evolution of a PDE. Existing methods employ values at different spatial positions at each time step, causing a long simulating time. LE-PDE accelerates by learning a low dimensional latent representation, and evolve the low-dimensional state, rather than the high-dimensional original variable. Optimization is performed upon reconstruction loss of the latent state and the evolution accuracy. Empirical advantages on time cost achieved over previous methods on different PDEs.
strengths_and_weaknesses: Strengths: 
Paper demonstrated empirical improvements on both 1D nonlinear PDEs and 2D Navier-Stokes PDE over multiple previous methods. 
Model has a relatively flexible structure.

Weaknesses: 
Did not study when can PDE states be encoded into a lower dimensional state, and.or how many dimension can be get rid of. 
Did not study the number of parameters used against other methods, using an extra encoder and decoder might take extra number of parameters, resulting in unfair advantages over other methods.
Figure 1 is unclear. Specifically, is the box on the left the initial condition and boundary condition of the PDE? The schematic in the middle is also unclear.
questions: It would be interesting to see a study of when can the proposed LE-PDE method have a reasonable result. Does the reconstruction take extra data to train? 
Also, there has been numerous dimension reduction techniques, PCA, VAEs, just to name a few. Why did the author pursue the proposed method rather than making use of the existing ones? Are there any theoretical relationships between the proposed method and the existing ones?
limitations: The author proposed to accelerate evolution of PDEs by first reducing its dimension. 
The idea is intuitive and straight forward, and is proven empirically with different PDEs. 
However, a theoretical study on what PDEs this reduction can be efficiently applied to is not conducted, and it is not argued why the proposed reduction method is better than existing ones like VAEs. 
The method is flexible, and should be able to be accompanied with existing surrogate models and other method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a neural PDE solver based on learning how to represent the system with an evolving latent global state.
The advantage is reducing the cost of approximating PDE solutions without significantly reducing accuracy.
The method can also be used to solve some inverse problems.

strengths_and_weaknesses: ## Summary (and strengths)

* Originality. The paper proposes an approach which is simple to understand and very general (allowing any architecture to be used for the various components e.g. encoders, decoders). Furthermore, it appears novel to me, although I'm not familiar with the neural PDEs literature.
* Quality. The proposed method is always at least competitive with existing methods, and outperforms them in some important examples. The experiments are thorough, covering all the important ablations I could think of while reading the paper.
* Clarity. Overall the paper is clearly written and easy to understand, but there are some minor problems discussed below.
* Significance.  The authors make the case for the importance of neural PDE solver well. 

In my opinion this is a good paper and clearly above the acceptance threshold.

## Weaknesses

I found few important weaknesses in the paper. Some minor points are listed below and in the "Questions" section.

While the paper is generally well written, there are a few typos and confusingly worded sentences.
Most of these are unimportant, but below I list a few which made the paper harder to understand:

* Line 75 "flow probe" -> "flow to probe"?
* Line 89 "location... that satisfies given" Remove "that satisfies"?
* Line 122 "LE-PDE relieves of local evolution" I don't understand "relieves" in this context - use another word?
* Line 205 "The prevents the gradient to pass through to the boundary parameter $p$ such as continuous location." I'm struggling to parse this sentence, though I think the point is that it's not possible to backprop through discrete variables.
* Line 236 "$\eta \in [0,0,2]$" Should this say "$[0,2]$"?
* Line 324 "average amount of the advected smoke simulated by the solver" I don't follow what this means and I can't find the exact details in the supplement.
questions: ## Major

* In the objective defined in line 127, is the right hand side missing integration over $x$? If not, then there is a different objective for every $x$ location. How are these combined? (Also, less importantly, what does the subscript $d$ refer to?)

* In Table 1, why is the WEBO5 accumulated error so large if this is the ground truth method?

## Minor

* As someone not familiar with this field,
I found the subsection on "Deep learning-based surrogate modeling" reviewed autoregressive methods very clearly.
However the 1-sentence summary of "neural operators" was too brief for me to understand.
Can you expand on the details of "neural networks that approximate a mapping between infinite-dimensional functions",
as this seems non-trivial to do.
* Line 107. In what sense is $f$ a partial function?
* Table 1. Why is runtime for FNO not included?
* Line 233-234. The text says you start with $n_t=200$ then downsample. But downsampling is to $n_t=250$, which is a larger value!
Is there a typo here? If not, why not keep the original value of $n_t$?
* Line 314: "our LE-PDE's ablated version without latent evolution". Can you describe this in more detail?
I couldn't find a description in the main paper.
(Maybe it's somewhere in the supplement - in which case adding a reference would be helpful.)
* Line 354 "Increasing M... will be countered by less number of examples (since having to leave room for more steps in the future)"
I don't understand this, can you explain in more detail?
limitations: The paper covers technical and ethical limitations clearly in the supplementary material.
I agree with the authors' assessment that there are no obvious negative social impacts.
I also appreciate that the authors included experimental results where existing methods (slightly) outperform the proposed approach.
This helps readers get a clear picture of the relative strengths of the methods in different scenarios.
review_title: null

Meta-review: metareview: The paper presents a new method for accelerating the simulation and inverse optimization of partial differential equations (PDEs) of large-scale systems. The proposed approach learns the evolution of dynamics in a “global” latent space (i.e., with fixed dimensionality). The reviewers agree the proposed approach is novel and empirically competitive. issues regarding experiments have largely been addressed by the authors in their rebuttal. Their authors are expected to add some extended discussion (if possible) on (theoretical) properties of PDEs where their approach is expected to succeed. Some of the reviewers increased their scores after the rebuttal period. 
confidence: Certain
Decision: Accept