Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn the exponential operator in time evolution using the Pade approximation. The proposed method shows the states of art results on 1d Burgers, KdV equation, as well as the covid 19 benchmark.
main_review: The paper proposes a novel and natural model for operator learning. 

Pros:
1. The paper writes out the time update as in Table 1.
2. The paper uses the Pade approximation for the exponential operator.
3. The paper gets the state-of-art results on both PDEs and real-life benchmarks.

Cons:
It seems to me that the exponential operator only addresses the linear term exp(tA), while the most challenging and interesting part of the non-linear equation is the time integral term addressing the non-linear term N. This design may have a limitation on harder non-linear equations such as the Navier-Stokes equation. Would it be possible to add an additional term to approximate the time integral?
summary_of_the_review: In my opinion, this work is novel and meaningful. However, it is less satisfactory not to address the more interesting non-linear term N. I recommend borderline acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new approach for solving the operator map problem which is based on the non-standard form and its approximation via exponential operators. The authors integrate their method in the recently introduced neural operator framework. The technique is evaluated on two synthetic PDE problems and one real life example. In all cases, the proposed approach improves over the other baselines on the relative L2 metric.
main_review: The recently introduced neural operator framework exhibits several advantageous properties, and works that extend and generalize these methods are of interest. In this context, the current paper proposes to model the inherent nonlinearity in the data via exponential operators. From a differential equations viewpoint, this is a reasonable assumption as many IVPs could be solved using exponential maps. The theoretical stability guarantee is a nice addition, especially since RNN are known to be challenging to train. Finally, while the evaluation is not extensive with respect to state-of-the-art baselines and datasets, it does convey the message that the proposed method attains SOTA or potentially even beyond SOTA results.

My main concern regarding the paper and the reason for my lukewarm score is the unclear novelty in the method. Specifically, the expRNN (https://arxiv.org/abs/1901.08428) and dtriv (https://arxiv.org/abs/1909.09501) architectures considered the exponential map in the context of Lie groups. Essentially, these approaches use the scaling-squaring and a Pade approximant of Al-Mohy & Higham for which an analytic Jacobian can be derived and computed. In particular, no matrix inversion is explicitly calculated (see https://github.com/Lezcano/expRNN/blob/master/expm32.py). Also, this matrix exponential computation is *not* recurrent, and its stability is mostly governed by the modulus of the eigenvalues of the generating operator ($\mathcal{L}$ in your notation). Given the expRNN/dtriv line of work, it is not clear why the authors design a new recurrent architecture for computing the matrix exponential, and not simply using expm in torch. This approach will eliminate the recurrence, use less weights, will probably be faster, and will actually compute the matrix exponential. Regarding the last bit---since your approach is based on a learning component whose role is to learn the inverse polynomial, it is not entirely clear what is the actual space of operators your method learns. Please clarify.

Minor comments:
	- Is $\tau$ fixed in the operator problem? Why, and why is this problem so significant? Typically, one is interested in the prediction of one or more steps to the future (as in your COVID-19 example)
	- Is your $T$ finite- or infinite-dimensional? It seems like it is infinite-dimensional, and thus some discussion is required regarding its approximation as the $A_i, B_i, C_i$ are finite-dimensional.
	- I am somewhat puzzled about Eq. (4). Specifically, $T$ appears on the left and right sides of the equation. In addition, some of the operators are not specified ($P_i$?). How is this form used in practice?
	- Why $A_i, B_i, C_i$ are modeled as exponential operators?


summary_of_the_review: The proposed method is interesting as it extends a recently introduced framework (neural operators). However, the main technical novelty seems not necessary in the context of previous work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper study the problem prediction in time-evolving partial differential equations. 
Inspired by the nature of solutions in PDEs where the solution often time can be written in terms of exponent of the operator and Pade approximation exponent of operators, the authors propose a recurrent articture to learn solution operator in PDEs.

The paper proposes a nice idea and approach to solve the mentioned problem. 
main_review: The idea in this paper is novel and promising. The approach follows the basics in PDEs, resulting in an efficient method. 

There are multiple issues with this paper that I can not recommend acceptance at its current stage.
The first is related to the notation and explanation, and the second is the experimental study. 

1) It seems there is room to improve notation and explanation.
1a) In eq 3, what does the O plus sign do?
1b)  Is L an operator between function spaces or finite spaces?
It is not clarified.
1c ) the problem construction is missing in the paper. 

1d) how from Pade approximation on z in C do we have derivations of A applied on opeators? 
1e) This paragraph "Pade ́Approximation Givenananalyticfunctionfpzq" needs clean up. The sentences are not easy to follow.

1f) what are the norms? what are the input-output spaces of operators? what are the spaces?
1g) In the proof of Theorem, A is applied on z and also L. Is A a spectral function? 
1h) If everything here is finite-dimensional, then a more clarifying notation and problem structure are needed. It seems the authors jump between operator between function spaces and finite-dimensional vectors. The authors are encouraged to be consistent. 
1i) what is F in the proof of the theorem? 
1j) fix eq 19
1k) This list is longer, and the authors are encouraged to improve their problem construction.
1l) For what case Pade's approximation is valid? what are the a and b coefficients when dealing with operators?
2) The empirical study is not extensive compared to the prior works.
2a) are the numbers in the table and plots for one-step prediction? if yes, what happens if the operators are composed to predict future time steps?
2b) how the baselines are trained?
2c) Prior works study 2d Navier Stocks, the authors are encouraged to provide an extended study on that front to provide more compelling results on the benefits of their methods.
2d) study of further statistics is missing
3) Why not directly learn the exp(L) instead of using such thing as Pade approximation?

summary_of_the_review: A great work, and a great idea. 

However the notation needs a lot of work, the explanation needs a lot of work, the empirical study requires more work and explanation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a recurrent Pade network fro learning non-linear operator approximations for IVP.  The Pade exponential operator uses a recurrent structure with shared parameters to model the non-linearity compared to recent neural operators that rely on using multiple linear operator layers in succession. The paper showed that Pade network does not suffer from the issue of gradient explosion and the boundedness of the gradients can be established
main_review: This paper uses a rational polynomial to approximate the exponential function and proposed a Pade network. The idea is reasonable and the experimental result is convincing. Here are several minor points.
- The Pade approximation need the inverse, the paper use  a non-linear layer which we implement as a simple fully connected layer to approximate the inverse. Can the author justify this approximation, as an example justify the network is powerful enough?
-exp(x) can also be approximated by (1+(1/n)x)^{n}, can the author justify the benefit of the Pade approximation beyond the simple (1+(1/n)x)^{n} approximation. (the approximation doesn't need an inverse) I guess [1] is doing such approximation.
- After learning the operators in the Pade approximation, can the learned operator be generalized to other selections of (p,q)? As an example using one set of (p,q) for training and another (p,q) for testing?
- To justify an operator is learned, an experiment to show the generalization ability cross different grids is needed.

[1] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew
Stuart, and Anima Anandkumar. Markov neural operators for learning chaotic systems, 2021.
summary_of_the_review: My first main concern is that exp(x) can also be approximated by (1+(1/n)x)^{n}, the paper should take this as an baseline. Second is the nonlinear layer is not powerful enough to approximate the inverse.
review_title: null

Meta-review: The paper provides a unique contribution that uses Padde approximations to approximate non-linear operators for solving initial value problems in PDEs. The paper contains also a non-trivial experiment with a real-world dataset that showcase the impact of the proposed model. The authors have provided a strong rebuttal and therefore I recommend Accept.
Decision: Accept (Poster)