Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper suggests novel measures for evaluating latent dynamics model by measuring how close the latent dynamics is to being governed by a symplectic map, a property of Hamiltonian dynamics. The paper analyses variants of the original Hamiltonian Generative Network [48] using the model and identifies better variants. The measure is also compared to related measures such as the MSE image reconstruction error, or the valid prediction time (VPT).

main_review: 
Strengths:
- The proposed SyMetric measure is novel and can be interesting for theoretical analysis of learning latent dynamics in simulation environments.
- The identified improvements over HGN are a secondary contribution.

Weaknesses:
- Sec 2, l74 ff is not well comprehensible. A differentiable map F is proposed whose symplectic property is defined. No relation to the Hamiltonian is made. The last sentence in this subsection in l. 78/79 uses the term "symplectic map" for ds/dt without defining it properly. Later in l. 93 suddenly canonical transformations are mentioned as being “symplectic maps” without explaining why. Since these notions are at the core of Sym measure, they need to be defined clearly so they can be taken up again when introducing the measure. Please clarify.
- The Sym measure requires training a neural network for converting the learned latent state into the ground truth state representation. This is problematic for two reasons: a) the proposed measure is only of theoretical relevance, because it cannot be used in practice when ground truth is not available (otherwise learning a latent state representation would also be useless and the latent state mapping of the images could be directly learned onto the ground truth state representation). b) The analysis depends on the quality of the learned mapping which is unclear and can be prone to model errors. 
- Eq 2 has a sum ranging over t but it’s not used for any term inside the sum. Above the equation, the distribution p_\theta(x_t) = d_\theta( q_t ) is mentioned but never used again. What is the argument q_t of d_\theta ? What is p_\theta( x | s ) in eq 2 ? 
- The paper lists improvements over HGN but does not give insights why these changes make the model better.
- l. 119, how can a 1D phase space make sense, shouldn’t it be at least 2D (1D position and 1D momentum)? Can a 1D phase space represent all the environments which are evaluated in Sec. 7?
- l. 122 what does it mean to train the network on a “prediction” rather than a “reconstruction” task?
- Eq. 3 what is the definition of \hat{s} ?
- l. 224. The R^2 measure is motivated as goodness of fit, but it does not follow the usual definition \sum_i (F(S_i)-E(s_i))/E(s_i). Instead, it seems to compare the variance of mapping F(S) with the variance of the ground truth. The paper does not explain clearly, why this should be a meaningful measure. Why is a low number indicative of bad model performance? What happens if the learned mapping predicts larger or lower variance than the ground truth ? The motivation of thresholding for a low R^2 in l. 262 cannot be followed.
- The proposed SyMetric measure is an ad-hoc combination of the R^2 and the Sym measure and relies on an empirical choice of two thresholds. What does it mean for the thresholds to “work well”? How to choose them subjectively?
- What do the colors of the curves in the B plots mean? If the curves are prediction vs ground truth it seems the learned mass-spring +c clearly gets out of sync which is also detected by VPT, while the proposed SyMetric measure wrongly finds the learned model to mimic the ground truth dynamics well. Please discuss.
Overall the technical description of the proposed measures seems flawed which make the method difficult to comprehend. The experiments do show issues of the proposed metric where the baseline VPT method seems sufficient and superior. The paper does not seem ready for publication yet. 
 
  

Further comments:
- The description of HGN in sec. 3 should cite the original paper [48]
- def of VPT: please use “\operatorname{arg\,min}” 

== Post-rebuttal comments ==

The author response addresses several of my concerns well. I still rate the paper borderline ("marginally above acceptance threshold"), since the approach requires to learn a mapping from the learnt latent state space to ground truth which could be difficult to assess in its quality itself and can invalidate the proposed measures if the mapping is "off". The paper should provide an adequate assessment of this potential problem, for instance, suggesting ways to analyze the accuracy of the learned mapping and its impact on the proposed measure. 
limitations_and_societal_impact: The authors mention several limitations of their work in the discussion section. Further discussion is needed according to the limitations discussed in sec "weaknesses" above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper mainly focuses on the application of Hamiltonian Dynamic in computer vision tasks. The authors propose a new metric to evaluate how the learned latent space coincide with the ground truth. Based on the novel metric, they further improve the previous HGN with multiple modifications in different aspects, leading to much better performance.
main_review: 1.	The authors provide a comprehensive analysis on the drawbacks of MSE as measurement for the corresponding task.
2.	Rather than considering image-level metric, the authors take into account the canonical transformation for Hamiltonian dynamics, which is interesting and novel.
3.	The proposed method is proven to be useful when considering HGN as baseline.

limitations_and_societal_impact: 1.	The main contribution of this paper is the new evaluation metric SyMetric. However, two things which I think are important for this metric are not explained in the paper.
a)	The final design of SyMetric is a binary indicator function. In this way, how to rank multiple models when only SyMetric is provided without $R^2$ and Sym?
b)	Although the authors conduct extensive experiments on a bunch of dataset, all results are based on the method of HGN and other variants. Is there any chance that the proposed metric cannot generalize to other type of methods? I suggest extra experiments on works like [8, 44, 2, 57] to further prove this.
2.	Maybe it is better to place the introduction of HGN++ after the new metric is explained for better understanding.
3.	The authors mention that the design of HGN++ is selected using ‘proposed new measures’ (L106-107). However, it is not clear that:
a)	Which measurement is adopted during selection, Sym, $R^2$ or SyMetric? As I can understand, the HGN++ is designed so that one of these three metrics is consistently better. As explained in this paper, Sym and $R^2$ must be used together for comprehensive evaluation. However the proposed SyMetric is binary as mentioned above, thus unable to select among multiple choices. Moreover, the Sym of HGN++ is not consistently better than HGN among all datasets as shown in Tab. 1 which means the model selection may not be conducted based on Sym. 
b)	What strategy is used in the model selection? How to find the best combination of the different modification?
4.	Is it safe enough to get the claim ‘SyMetric is in fact correct’ (L305) based on one sample visualization in Fig. 2B?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a distance measure to measure the symplecticness of the Hamiltonian Graph Networks. The experiments show that this measure is rarely equal to 1. However, it is only True for models that generally perform good on the task.
main_review: I already reviewed the paper for ICML. The paper has been improved and the minor aspects were addressed. However, the main shortcomings have not been addressed (but most likely also cannot be addressed). The main shortcomings are:

* the similarity measure is too complex to compute. It requires an additional function approximator to compute the measure. Furthermore, this fitting is sensible to the choice of the black-box function approximator.


* the similarity measure is binary and only positive for two environments. Therefore, it is very hard to evaluate whether the distance measure is of any use or provides any intuition. It simply just shows that the currently learned latent codes are not that great.

To be honest, I do not think that there is any path to fix these two shortcomings without fundamentally looking for a new approach and evaluation. The approach needs to be improved such that the results increase my confidence in this measure as it is really hard to interpret. Due to these shortcomings, the approach lacks generality and I do not believe that the approach will make much of an impact on the community. However, the execution of the paper is very well done. The paper is well written, has nice figures and the evaluation is done thoroughly. Therefore, one cannot complain about the paper except the significance of the results. My personal take-away is that this paper provides a good documentation of performed research which leads to non-significant results. Therefore, I believe that the paper is valuable because of the good documentation, but its not significant to be presented at NeuRIPS. This conclusion is subjective and hence, please take it with a grain of salt. If the other reviewers think that the paper provides significant results please ignore my review.

### Post Rebuttal Comment:
The comments of the authors clarified many points. I still cannot champion the paper for acceptance. As there is nothing wrong with the paper and its mostly about contribution and impact, I am happy with a weak accept. However, this rating is only marginally above the borderline. Therefore, I will not argue for the acceptance. 

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper has three reviewers, and the initial review are two borderline reject, and one borderline accept. After very long and detailed rebuttal and discussion process between authours and reviewers, two reviewers had raised their scores. In general, this paper has a theoretically grounded working solution to a valid and important problem in the field. The negative points/drawbacks come from that the results are not convincing and partially negative results which are very hard to impossible to support empirically. And the  approach needs to learn a mapping from the learnt state space to ground truth which is difficult to assess in quality itself and can invalidate the proposed measures if the mapping is "off". Thus, the initial meta reviewer would tend to borderline acceptance. 
Decision: Accept (Poster)