Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The manuscript proposes a way to learn Hamiltonian systems with additive dissipation from (scattered) observations.
The learning is framed as a (Bayesian) inference of a Hamiltonian endowed with Gaussian Process prior with the rest of the generative model relating trajectories to the hamiltonian via dissipative dynamical equations, and noisy observations on top of these trajectories.

The prior covariance matrix on trajectories is approximated using RFF for an ARD base kernel and propagating the approximation via the linear operator specifying the dynamics.

Variational inference under this (approximate) prior model is used, which conveniently allows to use the reparametrization trick to evaluate the ELBO.

The resulting method is evaluated on a number of experiments and compared to pre-existing methods.

strengths_and_weaknesses: -- Strengths

The method is a useful extension of the SymGPR method to dissipative systems.

The combination of the prior on hamiltonian, dissipative dynamics and RFF appears original to me.

The combination of many different approximations (RFF, VI, stochastic evaluation) leads to a practical algorithm for the problem at hand.

The possibility to make predictions without or with different dissipation is expected but neat.

The results are impressive especially in the low data regime, where the method clearly outperforms alternatives in its prediction accuracy.

-- Weaknesses

I report here a few points that made the manuscript a bit difficult to read.
It is mostly about the motivation rather than the technical content.

1) Previous work, especially the symGPR is not introduced which makes it difficult to understand the novelty here.
What did they do exactly? did they also use RFF to approximate the covariance or do the calculation closed form?

2) the many approximations introduced are not necessarily well motivated. Why do you do the RFF? Is it necessary?
 convenient for VI? for scalability? I have my guess but this needs to be more explicitely stated in the paper.
 
3) The consequence of the approximations introduced are not discussed.
Does the RFF preserves the symplectic structure? What is lost by approximating the prior does it bias the inference? if so how?
The same applies to using VI.


Because of these perceived weaknesses,
I set my score to weak accept. I m willing to change my evaluation if my concerns are adequately addressed

questions: I don't understand the following sentence:
"We used a block diagonal approximation of C so that each pair of basis functions shared the same covariance".
Is this mean field $q(C)=\prod_i^M q(C_i)$, or $q(C) = \tilde{q}(C)^K$. In both case, this has strong consequence on the inference.
These choices should be discussed.

limitations: None
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Paper propose a spectral representation of Hamiltonian. Demonstrated the procedure to solve the equations of motion given a set of noisy measured trajectories (line 202).

The learning process is on the parameters used in the formalism of the special Hamiltonian (line 224).
strengths_and_weaknesses: Strengths:
The formalism and learning process is very different from a traditional Gaussian process approach. Hence there is some novelty in this paper.

Back information is well given and pitch at the correct level of sophistication. e.g. Hamiltonian systems, Gaussian process and literature review.


Weakness:
The main mathematical framework is given in page 5 and 6. I find that these two pages of the paper is extremely difficult to follow. This is the main weakness of this paper. I explain the reasons why these two pages are difficult to follow:
1. Eq 5. w_m is drawn from a normal distribution. In line 224, it is said that w_m is a learned parameter. Perhaps something is not explained clearly.
2. Line 195: why p(f|w) is a Dirac's delta function? How do we get Eq 9?
3. Line 208: justification needs to be given for distribution of x being a delta function Eq. 11
4. There are undefined math symbols, e.g. Eq 17, b and C
5. Does Eq 17 contradicts Eq 5?
6. The physical meanings of equations should be explained better.


It will be much better if the author rewrite the manuscript to explain more clearly and in a higher level of what their method is trying to do. After a good intuition is given to the readers, the authors may dive into details of the mathematics.





questions: Please read the questions given in the section "Strengths and Weakness".
limitations: Discussion on limitations is given in a short paragraph towards the end of the paper. The authors propose to extend this formalism into high dimensions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a symplectic spectrum Gaussian process method. The method can predict systems whose dynamics follow energy conservation and dissipation laws from noisy and sparse data. The proposed method is a general tool and has the potential to use the design of kernel machines with prior knowledge in physics. Although the proposed method is solid in theory, the scenarios applied are relatively simple. In addition, some classic examples such as vortex-particle and gravitational systems should be compared with the recently proposed methods.
strengths_and_weaknesses: Strengths

This paper proposed a symplectic spectrum Gaussian process method for modeling Hamiltonian systems with additive dissipation. They derived a new spectral representation by incorporating the symplectic structure to handle energy dissipation and energy conservation systems. They also proposed a variational inference procedure that offers numerical integration of the ODE solver as a subroutine. Finally, they demonstrated some experiments on several physical systems to verify the accuracy of the proposed symplectic spectrum Gaussian process method. 

Weaknesses

The demonstrated examples seem to be cherry-picked, and recently proposed methods, such as SympNet, can also accurately predict simple Hamiltonian systems.
- Adding some citations of recent papers related to symplectic neural networks would be helpful. 
- The symplectic neural networks are divided into separable and non-separable. The current paper seems to discuss only separable Hamiltonian systems, which needs to be clarified.
- Limitations of the method need to be discussed.

questions: Overall, the exposition of the paper is pretty straightforward. However, I assume it to be difficult for readers who are not familiar with Gaussian process models. More specific comments:
- The equations of the predicted systems should be given in the main text.
- Is the order of the subgraphs in Figure 4 reversed.
- The introduction of the theoretical model could be more concise.
- The idea in Figure 2 is not clear. Can you show a clear schematic?
- Why do the other methods in Figure 3 perform well in their papers?
- The selection of some parameters should be discussed in detail. For example, time step, parameter size, data set size, etc.

limitations: I would love to hear some limitations of the proposed paper about failure modes that the authors might have encountered.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Many neural network-based methods for continuous-time physical systems have been proposed recently. These methods are based on ordinary differential equations and consistent with geometric structures. In practice, the available data is noisy and sparse. To deal with such situation, this study proposes to use Gaussian process (GP) consistent with geometric structures. Because tha data is composed only of observations (without derivatives), the GP uses the random Fourier features.

strengths_and_weaknesses: *Strengths*

To my best knowledge, this is the first study to tackle the modeling of the Hamiltonian system (with dissipation) by using GP. Like the great success of Hamiltonian neural networks (Greydanus et al., NeurIPS, 2019), this study potentially opens up a new research field.

This study tackled the observation noise, which is a crucial issue in this field, but has been ignored by most existing studies.

The impact of the sampling rate on the performance, shown in Figure 7, is interesting. The proposed method is superior especially in the case of the sparse sampling.

*Weakness*

The explanation about geometric structures is inexact. See (1) below.

The generality of the spectral representation is unclear. See (2) below.

questions: (1) At line 67, the authors stated that "whose covariance function incorporates the geometric structure (also called symplectic structure) for the energy conservation or dissipation laws". The geometric structures appearing in physics are not limited to the symplectic structure, but include contact structure, Poisson structure, Dirac structure, and so on. Moreover, the symplectic structure is related to the energy conservation law, but not to the dissipation law. Please introduce and discuss the geometric structure exactly.

(2) To deal with the case without the true derivatives, this study proposes the symplectic random Fourier features. The experiments demonstrated that the features worked well for pendulum and Duffing oscillator, which exhibit periodic behaviors and are easily captured in the frequency domain. The generality for non-periodic cases such as double pendulums is unclear.

I am curious about the impacts of noise level on the accuracy. The comparison methods, namely HNN and SymODEN, ignore noise. In the noiseless case, is SymODEN enough?

*Minor comments:*

The experimental setting is a bit unclear.
(a) The difference between HNN and SymODEN is unclear. HNN learns the Hamiltonian using a single neural network, and SymODEN learns the weight matrix and the potential energy using two networks, obtaining the Hamiltonian. Is this OK?
(b) At line 298, the authors stated "Since HNN, D-HNN and SympGPR require derivative observations for training, we used the finite difference instead." Wasthe finite difference used as an approximation to the derivative?

The images are shown in the reverse order of the captions in the leftmost and second leftmost columns in Figure 4.

Due to the inexact discussion about the geometric structures, I cannot accept the present paper as it is. In this regard, I believe that a revision of the text would be sufficient. Then, I will be happy to raise the score.
limitations: As discussed in Section I (and potentially shown in Figure 10), the extrapolation performance might be limited. The authors sincerely expressed this limitation. I believe that this limitation will not diminish the value of this study.
review_title: null

Meta-review: metareview: Learning from continuous-time physical systems when input data is noisy & sparse, and without access to time derivatives, is a hard problem. The authors propose a novel algorithm using Gaussian Processes, guided by physical knowledge. Reviewers agreed that the work was original. One reviewer raised concerns about the readability of the paper. The authors' responses will likely address most of those concerns. Other reviewers also suggested a number of improvements, which the authors took on board and will easily implement. Despite relatively simple experiment scenarios, this new algorithm demonstrated some advantages in the low data regime, where it improves on previous known algorithms.
confidence: Certain
Decision: Accept