Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The focus of this paper is to develop a learning framework to discover spatiotemporal PDEs from scarce and noisy data. 
The authors suggest devise a deep convolutional-recurrent network which encodes prior physics knowledge followed by 
sparse regression with reconstructed data to identify the analytical form of the governing PDEs. The framework is validated 
on three high-dimensional PDE systems where superiority over baselines is demonstrated.
main_review: Major comments:

-) The written English language needs to be improved -- the paper is reasonably written from a technical perspective but 
grammar is not on par.

-) In Section 4, how is the ground truth obtained?

-) One thing I find missing is wall-clock times. I can see that for the experiments and parameters reported in Section 4, the 
proposed algorithm returns better qualitative results. Nonetheless, how much faster/slower is the new method versus the 
competition?

-) My main concern is that the sparse regression step can be very slow due to the very large library \Theta(U). Isn't a pre-processing 
step like PCA needed in practice? The reason I mention this is because several columns of this matrix are not contributing much.

-) In terms of novelty, the sparse regression step is fairly well-known. I know of some work done on reconstructing noisy samples 
(the authors cite a good part of the related literature on the topic) but how exactly step 1 is really different from previous work? 
The authors discuss the topic but it's not really clear.

-) Unfortunately, the fine tuning step depends on that of sparse linear regression, i.e., the correct PDE form need be determined. 
The authors mention this themselves, but any guidance on whether the entire mechanism need be restarted (with different parameters) 
or some of the work can be salvaged would be interesting.

Minor comments:

-) "Recently, a ground-breaking work of sparse identification of nonlinear Dynamics (SINDy)..." Please capitalize as needed.
-) "Furthermore, the introduction of spatial derivative terms and improved the sparsity-promoting algorithm of sequential threshold ridge regression..." This phrase doesn't make sense.
-) "...many works attempt to relieve the pain..." Please re-phrase. 
summary_of_the_review: Overally, the proposed algorithm is discussed at a good abstract level, and experiments on a few small datasets are provided. 
See my comments above for more details.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on discovering the underlying physics dynamics, in the form of partial differential equation (PDE) from low-resolution (LR) measurements. It leverages sparse regression-based recovery to identify the constituents of the unknown PDE. The main contribution is a module that constructs high-resolution (HR) observations from LR data, for sparse regression to perform well.
main_review: - 1. Consider defining all terms in equation (1). Here, x, t, u_x, u_y. Also, notation such as \nabla^2 should be defined in a separate section (in the Appendix) and cited early on (nabla is first defined in section 4.1). Also, the acronym I/BCs is not defined. 

- 2. The product block (PB) seems to be central to the discussion but this block is not motivated in the manuscript. Why did the authors choose this architecture? What is the role of PB?
	- a. While eq. (2) utilizes u_k to form \mathcal{F}_hat(u_k), Fig. 1b shows mapping of u_k to u_k. This is pretty confusing. This can be mitigated by labeling 1b according to (2). In other words, where is (2) in Fig. 1b?
	- b. It seems that \mathcal{F}_hat(.) is this supposed to mimic action of \mathcal{F}(.), however it is unclear why it should take this form.
	- c. Essentially, it seems that Fig. 1b is trying to emulate the forward Euler time stepping, as opposed to this being the action of the overall network, since ISG generates HR from LR. If so, consider correcting para. 2 on page 4. 
	- d. The terminology "highway physics-based Conv" layers needs citation. 

- 3. In Section 3.3:
	- a.  u is defined as a n_s x n_t matrix while in in equation (1) it is a vector of size n. This is pretty confusing since the discussion so far considers u as a vector. Consider updating the notation in (1) by stating the more general spatiotemporal problem. 
	- b. Regarding the problem stated in (4) -- 
		- i) Why do the authors solve (4) via STRidge, when they can use Iterative Hard Threasholding (IHT) (Haupt and Nowak (2006), Blumensath and Davies (2009)) directly? Arguably, there is no need to solve the l2-regularized least squares problem first, which may lead to faster solutions. This choice needs to be justified adequately.  
         - ii) Also, (4) is the l0-regularized least squares problem. There is a rich body of literature devoted to l0-minimization, and needs citations to some iconic works, and potentially a section of related works devoted to it (in the Appendix).
		- iii) The paper claims that STRidge "iteratively searches for the optimal tolerance based on the selection criteria (4)". STRidge provides a sparse estimate for a given tolerance, and does not conduct a search. It may be that the authors conducted a search over various choices of the tolerances. If so, consider correcting this statement.

- 4. In Section 3.4:
	- a. u_tilde is a 3d tensor of dimension here. It is unclear why this is a tensor. Consider clarifying and also, define H' and W'. 
	- b. The discussion in "Data reconstruction" is very abrupt and does not tie-in with the discussion so far. This is exacerbated by the changing dimensions of u. How does this achieve interpolation? This weakens the entire discussion, so I highly recommend re-writing this part keeping the notation consistent. 
	- c. What is an "IC" regularizer? "IC" is not defined.
	- d. What are u_0 and u_tilde?
	- e. Does the sparse regression use the reconstructed HR observations? If so, should this use the italicized capital U notation?
	- f. Last para. pg. 5 -- "The obtained coefficients from sparse regression may not fully exploit all the available measurement as the regression is performed on subsampled data." -- In the discussion preceding this statement the sparse regression was performed on the HR observations. 
		- i) Why and when was the data sub-sampled?
		- ii) Why do the authors initially form the HR data when the Sparse regression is performed on the subsampled data?
		- iii) Also, this may be partly due to the use of hard thresholding on the l2-regularized least squares. The l2-regularization may not let the coefficients to scale properly. 

	- e. How are these components trained? Although individual components are described, there is no discussion on how these work together or the training procedure. I am assuming these components are trained separately? Consider adding a discussion about this.
	- f. The role of recurrent network in 1b is unclear. The various vertically arranged conv blocks result from the final u_k? Consider clarifying this. 

- 5. The authors claim that one of the advantages of the work is its interpretability. Is it due to the interpretability due to the sparse coefficients being able to identify the components of the pde? If yes, then:
	- a. there should be a discussion of the interpretability properties.
	- b. the experiments should demonstrate how the recovered coefficients and the  

6. I recommend adding a baseline solving l0-regularized least squared directly using IHT by removing the l2 regularization. This is essential to analyze the performance of the proposed method. 

References:
- HAUPT, J. and NOWAK, R. (2006). Signal reconstruction from noisy random projections. IEEE Transactions on Information Theory, 52 4036–4048
- BLUMENSATH, T. and DAVIES, M. E. (2009). Iterative hard thresholding for compressed sensing.
Applied and Computational Harmonic Analysis, 27 265–274
summary_of_the_review: Although the paper proposes these modules and tries to solve an interesting problem, it is unclear how these modules work together. Also, they use a sparse regression method (STRidge) which seems like a round-about way of solving the problem directly, which may potentially be faster. The paper is well-written in parts with some notational issues and incomplete description. That being said, I hope the comments are useful for updating the manuscript in the response period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Problem: data-driven PDE discovery methods are not robus to low-quality measurement data.

Solution: A novel architecture that can encode prior knowledge (known terms, PDE structure, boundary conditions), and sparse regression procedure that is hypothesized to be more robust to scarce and noisy data scenarios.

Other Contributions:
* proof that proposed Pi-block is a universal polynomial approximator. This is why I scored a "4" on technical contributions.
main_review: Strengths
1] Procedure is well-defined/described*; Clean figure 2. The method makes sense to me.

Suggestions


1] At first reading, I had a hard time parsing the 2nd contribution bullet point. I was thinking "Is sparse regression the contribution? The inheritance?  Fine-tuning? Or the actual progression through these stages?" After reading, I see what you mean, although instead of 'inheritance' maybe you should say simply how you put it later in the paper "In the fine-tuning step... measurements are used to train a recurrent block completely based on the identified PDE structure from the sparse regression."
* The other thing that took me a second to understand was the recurrent block fine-tuning. I think I understand now that you are scrolling through time computing u_k+1 from u_k using the block from Fig 3b. It may be useful to put the Fine-Tuning loss function -- is it the full vector ||u - U(initial)|| ? Or tuning at each time step before going to the next?

2] I suggest to boldface the winning values in each column of Table 1.

3] The following notes are why I scored a value of 3 on "correctness of claims" and "empirical significance":
When I see such a huge improvement over the baseline, ...
a) I wonder if there is a better comparator worth considering? Unfortunately I am not well versed enough to suggest something specific.
b) I would like to see a progression of difficulty starting from when the methods do approximately the same (e.g., HR non-scarce data) to when a winner emerges (LR + scarce) 
summary_of_the_review: The paper presents a clearly described and sensible approach to PDE discovery. The main problem they seek to solve is robustness to data quality and scarcity, which they address with a combination of sparse regularization, fine-tuning, and prior knowledge encoding architecture. I am not extremely familiar with the field but I believe, especially the latter technique, is quite novel. My only real complaint is that only simulation data was used, and it's not "stress tested" in the experiments or against competitors in terms of practical/empirical significance (and hence fully supporting the claims of robustness). Note: I scored myself as a 3 because I'm not that familiar with the related literature.


Edit: My confidence on this topic is low because I do not know the literature a priori. However, I am not convinced by Reviewers x24u, zdB8 that the Pi-block is redundant with PDE-Net (to me it seems more efficient and sparse, at least), and I call into question Reviewer zdB8's claim that this is a "more complex" system than DLGA-PDE. All in all the other reviewers have shaken my confidence in the empirical results and rigor is methodology descriptions, but not as much in the novelty. I will lower my score to 6 to reflect this.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a physics-encoded discrete learning framework to improve the robustness of the PDE-FIND algorithm. The physics knowledge such as known terms and initial conditions can be used to reconstruct higher-resolution data. Then, sparse regression is performed with fine-tuning of coefficients to finally determine the discovered PDE.
main_review: The authors make substantial clarification and revisions upon the reviews. Especially, the product operator in the proposed block is indeed expanding PDE-net's capability. Based on these, I raise my score to 5, although still not meeting the acceptance bar. The authors may need more time to revise the manuscript but it has been better than before. The main concern is experiments, which need more time. The presentation also needs time for reconsideration and refining. I raise my confidence from 4 to 5 based on the discussion with the authors.

-------------------------------------------------------------------------

○ Strengths:
1. The ability to reconstruct higher-quality data by available physics data.
2. Promising results over PDE-FIND (Rudy, et al. 2017.) on the selected equations.
3. A very thorough review of prior works.

○ Weaknesses:
1. The novelty is limited. The authors propose to reconstruct higher-fidelity data via a recurrent convolutional network, which is straightforward but not very novel. Many image resolution recovery papers have used similar techniques of Deconv to increase resolution. In Eq. (2), Conv filters are used to process data, while there is no indication of how physics information is encoded. In Eq.(5), the loss is computed by both u and u0. It is not sure how and what physics is encoded into your system as claimed in the introduction. Could you explain?
2. Confusing presentation. Many notations are not clearly stated. For example, on Page 5, the authors write both x˜ and u˜ as measurements, which is confusing. On page 3, x and t are not introduced. From the paper, readers can hardly interpret their meanings. What is u_y if y is not an input variable in Function F? It is confusing to put u_{k+1} and u_t together, as the first one means time step while the latter one means time derivative. Moreover, what is the meaning of the dimensionality of u (1xn)? Which one is the spatial dimension and which one is the temporal dimension? The definitions are not included or are vague. 
3. Only one baseline published in 2017 is compared in the experimental section, while many papers within the recent three years also claim to improve the robustness of PDE-FIND and solve the low-quality data problem, such as [1-4]. Thus, this paper’s improvement is not persuasive enough without comparisons to state-of-the-art models. 
4. This method solely discusses the discovery of PDEs with constant coefficients and discrete data and the selected three equations in the experimental section are quite simple compared to equations used in prior works, whereas many papers in recent years have moved forward to the discovery of more complex PDEs with continuous mesh-free data. Why not experiment on more complex PDEs like prior works do to adapt the real-world scenarios, such as the 2D Navier-Stokes equation? The influence of this work is limited if only discrete data and constant coefficients on simple PDEs are discussed. 
5. There is no demonstration of real-world data. The data used in this work is solely based on mesh-based simulation, while most real data do not have high-fidelity at all mesh-grid points, which means the data is not only noisy but also sparse. The authors only discuss the robustness against noise. How well does the method scale with sparsity? 
6. The noise range tested in this paper is limited compared to previous work. The authors show that their method is good with noise up to 10%, however, most prior works can also handle 10%. I would be interested to know whether the method can handle larger noise to compete with prior works.

[1] Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. arXiv preprint arXiv:2005.03448, 2020.
[2] Jun Li, Gan Sun, Guoshuai Zhao, and Li-Wei H. Lehman. Robust low-rank discovery of data-driven partial differential equations. In AAAI, pp. 767–774. AAAI Press, 2020a. 
[3] Hao Xu, Dongxiao Zhang, and Junsheng Zeng. Deep-learning of parametric partial differential equations from sparse and noisy data. Physics of Fluids, 33(3):037132, 2021.
[4] Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time PDEs from sparse data with graph neural networks. In International Conference on Learning Representations, 2021.

summary_of_the_review: The paper is technically correct but lacks novelty and clarity. The experimental demonstration is not sufficient to support that it can also work for state-of-the-art PDE discovery models and for more complex PDEs. The reported robustness is also not exceeding the state-of-the-art models. Besides, the presentation and notation definitions can be improved. Based on these observations of the submitted paper, I lean towards rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a deep convolutional-recurrent network plus a sparse regression model for discovering spatiotemporal PDEs. The three-dimensional PDE experimenta l results are used to validate the effectiveness.
main_review: ### Updates after discussions
My concerns are partly resolved and I raise my score to 5. Detailed can be found in the discussion threads below.

Generally this paper is clearly-written and the PDE discovery problem is of great importance in the pioeering research fields. This paper uses some deep cov-net and sparse learning techniques to learn the promising results. 

### Main concerns
The main concern of mine is the novelty and experiments.

* PeRCNN is proposed in the paper to generate HR data from LR data. However, this method is similar to the existing idea of directly building a neural network to generate meta data, such as DL-PDE[1]. So how much does a more complex network model (i.e., PeRCNN) improve the performance? In the experiment corresponding to Figure 2 and Table 1, the authors should consider comparing the performance of PeRCNN with the method of using a simple fully connected neural network to generate HR data. In fact, many algorithms based on sparse regression or genetic algorithms are already optimized for LR data in PDE discovery. 

* The authors are suggested to compare more existing studies to show the advantage of the proposed method. Although PDE-FIND is classic and effective, it is not robust enough to noise and requires that the candidate set contain the correct solution. Therefore, there have been many new attempts in recent years, and the author should compare at least some of these studies. For example, the DLGA that also uses NN to reduce the interference of noise on the model [1], the EPDE that uses genetic algorithm to expand the candidate set [2], the AI Feynman that based on symbolic regression [3], and the SGA that can mine open-form equations form data without prior knowledge of the function terms [4], and the KO-PDE that can discover highly nonlinear coefficients of the PDEs [5], etc. Besides, authors could discuss the difference between the Π-block and PDEnet/PDEnet2.0 [6, 7] in the data reconstruction process. 


* When faced with noisy data, there are two common ways to calculate the gradient/function term: i). Use neural networks to generate meta data, similar to the PeRCNN, and then use the difference method to calculate the gradients.ii). Directly use the automatic differentiation process of the neural network to generate the gradients (e.g. Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems). Both of these ideas are much better than directly differencing the noisy data. The authors should compare more models in the experiments, including the several studies mentioned at the end of the section 2 of this paper, and some other studies dedicated to sparse and noisy data. It is not fair or sufficient to only compare PDE-FIND with the proposed model since it is designed for HR data. The authors should consider more baseline models in the experiments.

### Other concerns
Besides, I am also concerned with the claims on knowledge, data and experimental settings:
* The authors claimed that “Notably, this network is characterized with the capability to encode given physics knowledge”; “The overall network architecture mimics the forward Euler time stepping”; “The recurrent Π-block is a major innovation of this network architecture to capture the dynamics”. What is the difference between PeRCNN and PDE-net and PDE-net 2.0 (using the kernel to represent the derivative term, and use the δt-block to mimic the forward Euler time stepping)?
* The author mentioned that “The obtained coefficients from sparse regression may not fully exploit all the available measurement as the regression is performed on subsampled data.”. Sparse regression is based on HR data, why is it subsampled? Why not use all HR data directly in the sparse regression, but perform additional fine-tuning? If the essence of fine-tuning is coefficient fitting, then the only influencing factor should be the amount of data.
* The author mentioned that “the dominant reconstruction part lacks explicit physical interpretability (i.e., grey-box model), which may induce the difficulty in optimization especially for high-dimensional PDE systems with delicate patterns.”. Why does the process of generating HR data need to be interpretable? It seems that the PDE discovery can be realized as long as the process of mining PDE from HR data is interpretable.
* The author should use the same candidate set in different PDE missions. It seems that the current candidate set is designed according to the PDE to be discovered. Besides, the number of candidates is relatively small, which reduces the difficulty of the problem. In practical applications, it is difficult to give a specific candidate set for a certain mission since we do not know the form of the PDE in advance. The candidate set should contain all possible candidates, and should not change according to the mission.

### Minor points:
There are two “the” repeated at the end of the ninth line on the third page.

### References
1. Xu, H., Chang, H., & Zhang, D. (2020). DLGA-PDE: Discovery of PDEs with incomplete candidate library via combination of deep learning and genetic algorithm. Journal of Computational Physics, 418, 109584.
2. Maslyaev, M., Hvatov, A., & Kalyuzhnaya, A. (2019, June). Data-driven partial derivative equations discovery with evolutionary approach. In International Conference on Computational Science (pp. 635-641). Springer, Cham.
3. Udrescu, S. M., & Tegmark, M. (2020). AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16), eaay2631.
4. Chen, Y., Luo, Y., Liu, Q., Xu, H., & Zhang, D. (2021). Any equation is a forest: Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE). arXiv preprint arXiv:2106.11927.
5. Luo, Y., Liu, Q., Chen, Y., Hu, W., & Zhu, J. (2021). KO-PDE: Kernel Optimized Discovery of Partial Differential Equations with Varying Coefficients. arXiv preprint arXiv:2106.01078.
6. Long, Z., Lu, Y., Ma, X., & Dong, B. (2018, July). Pde-net: Learning pdes from data. In International Conference on Mac
summary_of_the_review: This paper addressed an important problem, PDE-discovery and was clearly written. Some points are new but there are some critical issues on novelty, experiments, knowledge and data which suggests a reject.
review_title: null

Meta-review: The paper introduces a pipeline to discover PDEs from scarce and noisy data. Reviewers engaged in a very thoughtful discussion with the authors. I read the extensive rebuttal, and I believe the authors have addressed the major concerns claimed by the reviewers. I ask the authors to make sure to include all the changes and additional experiments in the camera-ready version.
Decision: Accept (Poster)