Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a manifold learning approach for speeding up solutions to parametric PDEs, inspired by DeepSDF [22]. They associate a latent vector for each each instance of the PDE, and learn a solution manifold using an initial dataset of PDE instances. During inference, they either search for the closest latent vector (using gradient-based optimization) (MAD-L) or fine-tune both the network parameters alongside the latent vector (MAD-LM).

They report strong performance improvements compared to Meta-Learning, PINNs from scratch, and DeepONets.
strengths_and_weaknesses: The paper attacks an important problem. Often times we have to solve many versions of a PDE, either for design optimization or parameter discovery. One of the strengths of using neural networks as an ansatz for PDE solutions is that we can use techniques such as meta-learning to speed up solution process. This paper adapts methods from the shape representation literature to this problem, and shows consistent performance improvements.

The contribution is simple but works well, so I think it is solid. I like the sections on the manifold learning intuition, especially the simple ODE example, as I think it really motivates why the algorithm works.

It would be nice to see results from larger-scale experiments. Could you demonstrate a case-study where you would want to solve a sequence of PDEs, and your meta-learning setup gives large performance gains? I think this would help motivate the problem of parametric PDEs much better.
questions: For the experiments with MAD, you had to introduce extra parameters in the NN to take the latent vector as input. Does this significantly increase parameter count? Could you give the number of NN parameters for "From-Scratch" experiments vs "MAD-L/LM" experiments?

It would be really nice to see a comparison of these methods to more standard numerical methods (such as FEM/spectral methods). it would be interesting to see if MAD-L can get close to or even surpass more classical methods in terms of performance.

Quality of life nits: Could you add descriptions of the figures into the captions themselves as well? It becomes difficult to scroll back and forth.
limitations: It was really nice seeing the cases where MAD-L failed. I think comparisons with more standard numerical methods could also make it clear where this method stands in terms of potential limitations compared to state of the art.

Potential negative societal impact was not assessed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for learning the solutions of parameterized PDEs, where the PDE takes a set of parameters as an input. The paper introduces a neural network architecture (MAD) which takes coordinates as well as a hidden representation, which is implicitly encoded for each PDE parameters. After training, the proposed method solves optimization problem either to 1) find the best hidden representation or 2) find the best hidden representation and the best model parameters (i.e., weights and biases simultaneously).
strengths_and_weaknesses: [+] a simple, but effective idea to extend learning PDE solutions to parameterized PDE settings

[-] novelty: learning hidden representations of parameterized PDEs in training time and exploring the latent space for finding the best hidden representation for the new parameter instances have been studied in reduced-order modeling context (Eq (7.3) in Lee and Carlberg, JCP, 2020), which can be considered as an equivalent idea of MAD-L. Although the context and the optimization objective (Eq.(5)) are different, the idea of exploring latent space for the best hidden representation of parameterized PDEs has been out there. 
questions: - how does the size of the latent vector affect the performance? how many samples (S1) would be required to learn a meaningful latent space?
- what is the computational overhead to solve the optimization problems in MAD-L and MAD-LM, respectively? (compared to other baselines)
- for the extrapolation task, how would the model performs if the testing parameters are generated in a following way: what if the training parameters are generated in a bounding box and the test parameters are chosen outside of the box. 
limitations: the proposed method, in general, is technically sound, but would be more appreciated if they provide more in-depth analysis on the experimental results, e.g., providing intuitions for setting up the size of the latent dimension, the size of the decoder, the choice of optimizers, the choice of parameter set-ups, and so on. I believe all these choices would be PDE-specific and it would be great if the authors could provide some general guidelines for making choices on those matters. Also, it would be great to see whether this method can be generalizable to more complex PDE problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose Meta-Auto-Decoder (MAD), a mesh-free and unsupervised deep learning method that uses pretraining by encoding PDE parameters to latent vectors. MAD has a manifold learning interpretation. Experiments show that MAD is faster than other deep learning-based methods.
strengths_and_weaknesses: Strengths:
1. Well-motivated and clear presentation of contributions
2. Well-validated experimental result

Weaknesses:
1. Many concepts are shared with Meta-Mgnet and moreover not empirically compared with Meta-MgNet
2.  Pretraining part is a bit unclear to me (detailed in the questions part)
questions: 1. When pretraining, what data is used? In paper, it says 'randomly-generated'? Does it mean both target and pretrain data are randomly-generated? What knowledge is shared among these data?
2. When comparing effienciency, is the pretraining stage been taken into consideration for MAD?
limitations: So far no.
review_title: null

Meta-review: metareview: The paper considers a meta-learning approach to solve families of PDEs (i.e. learning a solution operator). The main idea of the paper is to parametrize the solution of a member in the PDE family as being a function of a learned latent representation $z$ or that PDE. (Precisely, at a point $x$, they parametrize the solution as $f_{\theta}(x, z)$ where $\theta$ is globally learned, and $z$ is specific to a particular PDE.)
Beyond this, the parameters are fit as usual by minimizing some variational loss (e.g. $l_2$ error), and adapted to a new instance of a PDE as usual (by GD on an appropriately regularized $l_2$ loss). 

The idea is fairly straightforward. It seems to perform well on small-scale data (e.g. families of Burgers' equation), compared to training from scratch and DeepONets, but comparisons on more challenging datasets are lacking. The authors also pointed out that fine-tuning (i.e. specializing to a new equation) is where the method shines --- which is reasonable given the parametric form that is fit. 


confidence: Less certain
Decision: Accept