Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper used autoencoders to approximate different phases of a PDE solver.
main_review: The method proposed in this paper is novel and validated. Since this work is about accelerating the PDE algorithms, I would expect to read more discussions on how or why this method can be faster than traditional solvers by the ratio given by the authors (e.g., 40-50x faster). Besides, it would be more solid if the author compare between using different sizes of layers (number of parameters) in the autoencoders. How is the balance between performance, accuracy, as well as generalization ability, regarding to the size of the autoencoders? How to choose the layer size regarding to different problems or some performance requirement?
summary_of_the_review: I would like to see the acceptance of this paper if the authors can provide more comparisons and discussion on the performance / accuracy regarding to the size of layers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new ML approach called CoAE-MLSim that is a faster alternative to PDE solvers. Compared to previous ML work on this problem, it aims to be more accurate and generalize better across PDE conditions. They also aim to require fewer PDE solutions to train the model. 
main_review: *Strengths:* 
- I think this is an important application area with growing interest. 
- The approach seems smart to me. In particular, I could imagine this parallelizing well, it makes sense to me that applying the same network to each subdomain would allow for simpler networks, and it makes sense to me that learning low-dimensional representations with autoencoders could help. This is especially interesting to me in the context of learning a representation for the PDE conditions that you wish to generalize over. 
- My favorite part might be the dynamic inferencing with the iterative algorithm to establish consistency, enabling the networks to focus on subdomains. 
- Being able to initialize the ML solution with coarse-grained solutions from the PDE solver to speed up convergence is also intriguing. 
- I appreciate that the paper included results on quite a few problems, covering a variety of scenarios (steady-state vs. transient and generalizing across boundary conditions, sources, geometry, and initial conditions). 

*Weaknesses:*
 
I think that:
- The comparison to baselines could be improved. 
- Some of the claims are not carefully backed up. 
- The explanation of the relationship to the existing literature could be improved. 

*More details on the above weaknesses:*

*Comparison to baselines:*

"We did not find good benchmarks to compare our unsupervised, iterative inferencing algorithm against" I think this is a slightly unfair comment. The unsupervised and iterative inferencing aspects are only positives if they have the claimed benefits, as compared to other ML methods (more accurate and better generalization). There is a lot of recent work addressing the same ML task (as mentioned in the related work section.) This paper contains some comparisons to previous work, but as I detail below, there seem to be some holes.

FCNN is by far the strongest competitor for the Laplace example in the appendix. Why is this left off of the baseline comparison table in the main paper? Further, is there any reason that FCNN couldn't have been used for the other examples? 

Why is FNO not applied to the Chip cooling (Temperature) example?

A major point in this paper is improved generalization across PDE conditions. However, I think that's hard to check when only looking at the test errors for each method. In other words, is CoAE-MLSim's error lower than UNet's error because the approach fit the training data better, or is it because it generalized better? Further, in some cases, it's not obvious to me if the test errors are impressive, so maybe it is having a hard time generalizing. It would be helpful to see train vs. test errors, and ideally I like to see train vs. val. vs. test. 

For the second main example (vortex decay over time), looking at Figures 8 and 33 (four of the fifty test conditions), CoAE-MLSim has much lower error than the baselines in the extrapolation phase but noticeably higher in the interpolation phase. In some cases, it's hard to tell how close the FNO line is to zero - it could be that CoAE-MLSim even has orders of magnitude more error. Since we can see that there's a big difference between interpolation and extrapolation, it would be helpful to see the test error averaged over the 50 test cases but not averaged over the 50 time steps. When averaged over all 50 time steps for the table on page 9, it could be that CoAE-MLSim looks better than FNO just because of the extrapolation regime. In practice, someone might pick FNO over CoAE-MLSim if they aren't interested in extrapolating in time. Do the results in the table for vortex decay back up the claim that CoAE-MLSim is generalizing over initial conditions better than FNO, or is it just better at extrapolation in time?

*Backing up claims:*

The abstract says that the method is tested for a variety of cases to demonstrate a list of things, including "scalability." The list of "significant contributions" also includes "This enables scaling to arbitrary PDE conditions..." I might have missed/forgotten something, but I think this wasn't tested?

"Hence, the choice of subdomain size depends on the trade-off between speed and accuracy." This isn't clear to me from the results. It seems like 32^3 is the fastest and most accurate? 

I noticed some other claims that I think are speculations, not backed up with reported experiments. If I didn't miss something, this could be fixed by adding words like "might." 
- "Physics constrained optimization at inference time can be used to improve convergence robustness and fidelity with physics."
- "The decoupling allows for better modeling of long range time dynamics and results in improved stability and generalizability."
- "Each solution variable can be trained using a different autoencoder to improve accuracy."
- "Since, the PDE solutions are dependent and unique to PDE conditions, establishing this explicit dependency in the autoencoder improves robustness."
- "Additionally, the CoAE-MLSim apprach solves the PDE solution in the latent space, and hence, the idea of conditioning at the bottleneck layer improves solution predictions near geometry and boundaries, especially when the solution latent vector prediction has minor deviations."
- "It may be observed that the FCNN performs better than both UNet and FNO and this points to an important aspect about representation of PDE conditions and its impact on accuracy." The representation of the PDE conditions could be why, but it's hard to say without careful ablation studies. There's a lot different about the networks. 
- Similarly: "Furthermore, compressed representations of sparse, high-dimensional PDE conditions improves generalizability."


*Relationship to literature:*

The citation in this sentence is abrupt and confusing because it sounds like CoAE-MLSim is a method from that paper instead of the new method: "Figure 4 shows a schematic of the autoencoder setup used in the CoAE-MLSim (Ranade et al., 2021a)." More broadly, Ranade et al., 2021a, Ranade et al., 2021b, and Maleki, et al., 2021 are all cited and all quite related to this paper. It should be more clear how the authors are building on those papers (what exactly they are citing them for), and which parts of CoAE-MLSim are new. (The Maleki part is clearer in the appendix, but the reader shouldn't have to check the appendix to know what is new in a paper.) 

I thought that otherwise the related work section was okay but was largely just summarizing some papers without giving context for how they relate to this paper. 


~~~~



Additional feedback (minor details, could fix in a later version, but no need to discuss in the discussion phase):
- The abstract could be clearer about what the machine learning task is that CoAE-MLSim addresses. 
- The text in the figures is often too small. 
- "using pre-trained decoders (g)" - probably meant g_u? 
- Many of the figures would be more clear if they said pre-trained solution encoders & solution decoders, since there are multiple types of autoencoders.
- The notation is inconsistent, especially with nu. For example, the notation in Figures 2 & 3 doesn't seem to match the notation in Alg 1. Then on Page 4 & Figure 4, the notation changes again.
- Why is the error table not ordered 8^3, 16^3, 32^3 like Figure 9? The order makes it harder for the reader to reason about the tradeoff. 
- Why is Err(T_max) negative sometimes? Maybe I don't understand the definition, but I would expect to use absolute value?
- I don't think the study about different subdomain sizes is an "ablation" study since they aren't removing a component of the method. 
- Figure 11: I'm guessing that the y-axis is log error, but this isn't labeled as such. I didn't really understand the the legend or the figure in general until I got to the appendix, since there's little discussion of it in the main paper. 
- "Figure 30 shows comparisons of CoAE-MLSim with Ansys Fluent for 4 unseen objects in addition to the example shown in the main paper." - probably from previous draft. Now this whole example is in the appendix, unless I missed something. 
- My understanding is that each type of autoencoder is trained separately and that there's an ordering that makes sense to do this in, so you can use one trained autoencoder for the next one (i.e. train the PDE condition AEs, then the PDE solution AE, then the flux conservation AE, then the time integration AE). This took me a while to understand though, so maybe this could be mentioned in the body of the paper. (Or perhaps I missed that!)
- It seems that the time integration autoencoder isn't actually an autoencoder if it's outputting the solution at the next time step, not reconstructing the input. 
- Either I don't understand Figure 5 or the labels are wrong. 
- It's implied in the paper (like in Algorithm 1) that the boundary conditions are encoded like the other PDE conditions. In the Appendix (A.1), it's stated that "The training portion of the CoAE-MLSim approach proposed in this work corresponds to training of several autoencoders to learn the representations of PDE solutions, conditions, such as geometry, boundary conditions and PDE source terms as well as flux conservation and time integration." But then later in the appendix (A.1.3), it's stated that boundary conditions could be learned with autoencoders but are actually manually encoded for this paper. That seems misleading. 
summary_of_the_review: I think that CoAE-MLSim seems like a good idea and could be a great method. However, the results are not yet strongly backing up the claims, especially that it is more accurate and generalizes better than the previous ML methods for this task. I think that a revision during the discussion phase could raise my score to 8/accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propsed a composable AE-based PDE solver that combines traditional PDE solver with ML techniques. It split the big domain into sub-domains and compress the high dimensional representation into lower dimension using autoencoder.
main_review: Overall the clarity of the paper needs some seriour improvement. 

Questions/Weaknesses:
1. The related work sections covers a wide range of works that related to pde solving using neural networks. Although the paper provided a great list of related works, the relationship between this work and those mentioned is unclear to me. This is a critical piece that I am missing. If the related work section is delivered with more emphasis on the relationship instead of a mere list, it would be much improved.

2. The clarity of method description needs some significant improvement. Possibly rising from the size of the figures being too small, I had a hard time understanding the general flow and what to expect for the method proposed. A lot of questions like what is the dimension of each latent space, what is the size of your encoder network and do all the subdomains share the same encoder (I believe so)? How does the solution conditional AE take the source and geometry as conditions? Are they just appended to the input of both encoder and decoder?  

3. The autoencoders usually does not generalize well for the unseen data and the decoded result from unseen latent space is usually not valid in the original space unless being regularized properly (like Variational autoencoder etc.). How does this method do when facing out-of-distribution test cases?

Couple of minor points:
1. I believe the Figure 1 is not mentioned in the main text?
2. The figures are really too smallto see clearly. It heavily undermines the clarity of the paper
3. Page 4 alg 1 line 5 there is a error in latex? I believe it is <?
4. What is \THETA in page 4 alg 1 line 8? 
5. In Section 3.4 line 6 what is the citation [Ranade et al., 2021a] for? CoAE-MLSim is the proposed algorithm right?
6. For result part, again the size of the plot makes it impossible to read
7. The final table, there is no caption or table label and the unit is unclear
summary_of_the_review: The paper is lacking in clarity as mentioned above. Although technically might be novel and can benefit the scientific computing community, this is far from publishable from a perspective of readability. 

Update on Nov. 20th during rebuttal:
I think during the rebuttal period, I have gain much more understanding of the paper and willing to raise the score. My remaining concerns are clarity of the paper (including the fact that I still need to use 200% to see whats in the figures, but understandable given the page limit) and out-of-distribution generalizability.

The paper raised an interesting idea, using changing the simulation to the latent space by compression of autoencoders. However as the overall goal is still to simulate new designs, the generalizability of it due to autoencoders might still bottleneck its practicle impact.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed CoAE-MLSim to learn with relatively fewer samples of PDE solutions and solve PDEs. CoAE-MLSim uses the idea of domain decomposition: first learn the solution on local subdomains using autoencoder, and then couple these sub-solutions by an iterative algorithm. Numerical experiments are performed to test the efficiency of the method.
main_review: Strengths:
- The proposed method uses the idea of domain decomposition, and thus it has the potential to only need a small dataset to train the networks.

Weaknesses:
- The method requires 100-1000 PDE solutions for training, which is not really a very small dataset as they claimed. For a comparison, in the paper arXiv:2010.08895, it also uses 1000 solutions for the NS equation, and in this paper, it uses 500 in Section 4.2.
- The method part is not well written and hard to read. After reading Section 3.2/3.3, I only capture the main idea, but the detail of the algorithm is very vague and not well presented, e.g., how the neural networks are used here, what the network input and output are, how the flux conservation is applied, etc. The authors should rearrange Section 3 so that it is easy to read.
- During the inference stage, because the method needs to couple all the solutions in subdomains together by using an iterative algorithm. The inference time is significantly slower than other ML methods, which only requires one forward pass of the network.
- Also, it is not fair to compare the speed with Ansys solver, because Ansys is much more accuracy than the proposed method. If we need to compare with Ansys solver, then the authors should use a low resolution mesh of Ansys solver, such that it has a low accuracy similar to CoAE-MLSim, and then we can compare the speed.
- The accuracy is not very good. In Section 4.1, the MSE of the temperature is 5.3e-2, so without the square, the error is > 0.2, which is really large.
- For the result in Section 4.2, the timestep is dt = 1e-4, and here the authors only predict for 50 timesteps, i.e., t = 50 * dt = 0.005, which is too small. Even for the first few timesteps, the L2 error in Fig. 8 is almost 0.1, which is large.
- The authors claims that the method has a good generalization, but there is no evidence to support this. The prediction of future time in Fig. 8 is not really extrapolation. For example, we would not say time series prediction is extrapolation, and it is still interpolation in the space of PDE condition/solution.

Other comments:
- What does “CoAE-MLSim” stands for? Composable Autoencoder - Machine Learning Simulation? In the whole paper, “Composable” is only mentioned in the title, but not any where in the text, and thus it is not clear what it means exactly.
- In the abstract, the authors only mentioned the name of the method. After reading the abstract, I have no idea what “CoAE-MLSim” is.
- In the abstract, “unsupervised” is conflict with “relatively fewer samples of PDE solutions”. The method is not unsupervised.
- The Point 4 “The iterative inferencing algorithm is completely unsupervised and allows for coupling with traditional PDE solvers.” The iterative algorithm is part of inference, and the inference of networks of course doesn’t need the data. It is strange to say the inference stage is “unsupervised”. Also in Section 3.8.
- In Algorithm 1 Line 5, there is a “?”.
- There is empty space in the top right corner of Page 6.
- Is the SDF for the local domain or the global domain?
- In Fig. 4, the SDF looks strange. SDF is a continuous function, but in the figure, it changes from red color to blue color sharply and then keep the same blue color.
- The fonts in many figures are too small to read.

summary_of_the_review: The main issue is that their numerical experiments are not convincing enough to support their claims on small data, accuracy, and extrapolation.
review_title: null

Meta-review: This paper proposes a new approach called CoAE-MLSim as a faster alternative to PDE solvers.  According to the authors, the main strength of the method is that it needs fewer training data, however, as pointed out by the reviewers, the factor is not significant enough in the current results. Another concern raised by the reviewers is that the iterative algorithm is much more expensive than only one forward pass of other ML methods. Furthermore, the reviewers criticized that the comparisons with baselines are not sufficient, and some claims in the paper were not backed up. The authors provided their rebuttals and had a long discussion with most of the reviewers. Some clarifications have been made, and some reviewers increased their scores accordingly. However, overall speaking, the major problems with the paper still remain, and the rebuttal has not successfully convinced the reviewers to turn to the positive side. Therefore, we cannot give a green light to this paper yet.
Decision: Reject