Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper presents a generative approach to modeling physical systems with high-dimensional, nonlinear dynamical systems such as those found in fluid mechanics. The authors provide a physics-motivated hierarchical model for high-dimensional time series and a variational inference method for inferring latent variables and dynamical system parameters. They demonstrate its performance on simulated fluid mechanics prediction tasks.

Strong points:

The paper presents an interesting and motivating case for Bayesian inference in probabilistic generative models: a problem that has inherent uncertainty along with the ability to incorporate domain knowledge that can reduce the inference complexity.

The results on the simulated tasks are encouraging, especially the results in Appendix H which demonstrate generalization to out-of-distribution data. The paper does an excellent job of presenting and analyzing the results, along with compelling visualizations.

In general, the paper is well written (apart from some higher-level structural issues discussed below) and the notation is clear and unambiguous. 

Weak points:

Structurally, the paper could do a better job of motivating the particular choices made when designing the model. The choices that could be made more clearly motivated are:

1) using a complex-valued latent dynamical system. Although there is a sentence in the “Related Work” section (“the prior proposed and the use of complex variables enable the discovery of slow features”), this important choice could use additional discussion and motivation.

2) On first read, it was unclear what the likelihood model was, especially when it was stated that the observed data were time-series of (e.g.) particle velocities and positions. In Appendix E, it is stated that a Multinomial observation model is used, which could not be used for particle velocities/positions since it models counts of discrete variables. Thus, the observations $x_t$ in this case are statistics of velocities/positions (i.e. counts of particles in discrete buckets). This choice should be discussed in the main text, as it both changes the dataset size and dimensionality, along with the goal of the model itself. Rather than modeling particle positions and velocities, it seems like the model is generating statistics of the system. While this isn’t inherently problematic, this choice is not clearly indicated in the text.

3) Finally, the paper includes $z_t$ in the model to help make predictions converge to equilibrium. While intuitively, having $z_t$ decay according to the model corresponds to some notion of “stability”, the paper should also explain how the exclusion of $z_t$ can lead to unstable or diverging predictions. Although the paper cites related work to justify the choice, it is such an important choice that I think it merits more discussion in the main text.

These structural issues with the paper are worsened by the fact that there are no ablations or comparisons in the paper. I think the minimal set of additional experiments would include 1) a model with real-valued (not complex) latent dynamics and 2) a model without $z$ in it, to demonstrate the necessity of a stabilizing element in the model. Even better would be comparisons to non-probabilistic models, demonstrating the necessity of uncertainty in the face of “information loss”. Finally, comparisons to related methods (e.g. Koopman-operator baselines) would help understand how the proposed model improves on the surrounding literature.

Recommendation:

While I find the problem setting and proposed approach very interesting, the writing of the paper and the results still need work. The paper needs to better justify the modeling choices both in writing and with additional experiments. The provided results are encouraging, but I think a proper evaluation of the approach with the appropriate ablations and baselines is necessary for publication. I recommend a reject on these grounds.

Clarifying questions:

Are predictions obtained using MAP estimates of the parameters $\theta$ or by sampling the posterior predictive distribution (i.e. marginalizing out $\theta$)?
In what way is your inference approach “hybrid” as opposed to fully Bayesian?

Additional feedback:

Defining terms like “fine-grained” and “multiscale” would help make the paper more accessible to readers without physics backgrounds
Nit: “parametrized” -> “parameterized”
Page 4: “algrithm” -> “algorithm”


===========================================
Updates:

After considering the author's response and updates to the paper, I have bumped the score to a 6. The addition of Appendix H, in my opinion, considerably strengthens the paper's story and case for acceptance. I still have minor concerns about the writing surrounding the use of the Multinomial likelihood - for example, the paper still claims to be dealing with high-dimensional data, but bucketing into 25 buckets immediately reduces the complexity of the observation space to 25-dimensional counts. However, the authors have addressed most of my major concerns.

review_title: Interesting idea with encouraging results but lacks necessary motivation and ablations (updated)

final_score: rating: 7: Good paper, accept
review_content: This is a good paper, in my opinion. I have some suggestion to improve the quality of the work.
See below.

 - I suggest to improve Section 3, open a bit the range of references,  considering general and relevant works such as
 
C. Grigo et al. A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
arXiv:1902.03968, 2019.

G. Camps-Valls, et al. "Physics-Aware Gaussian Processes in Remote Sensing", Applied Soft Computing, Volume 28, Pages: 69-82, 2018.

Sungyong Seo et al. "Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics",  ICLR 2020,

- Explain better figure 2, improving its caption.

- Figure 3 is completely unclear, remove it or improve.

- Please explain better your system in Eqs (1)-(2)-(3). What are the measurements/observations? what is your inference goal? please clearer state these points.

review_title: Interesting paper. But some parts can be improved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a generative state-space model using two layers of latent variables. The latent variables in the first layer aim to capture long-term dynamics and ensures the stability. The variables in the second layer are physical variables. The authors have shown some promising results in modeling particle dynamics.  

Since the authors claim that the use of X can reduce the complexity/the search space of the learning model, it would be great if the authors can show how the performance change given different number of training samples. 

Another issue is about the effectiveness of this model in simulating real-world physical phenomena. This method can work well on simulated dataset because simulated data always follow the dependencies between X and x (as the simulator is built based on these rules). Real-world physical systems can be complex and usually we do not know exactly the governing physical variables (X). It would be great to discuss whether the proposed method allows some flexibility to automatically discover these unknown physical variables.

review_title: Nice way of combining physics

final_score: rating: 7: Good paper, accept
review_content: 
This paper proposes a new dimensional reduction scheme, based on a latent representation, using a generative model and variational inference.

The method is successfully applied to 2 simple one-dimensional physical processes, for which the macroscopic equations are known from the physics literature.

Choosing only a couple of simple physical variables X (as far as I understand, here it's the density in binned areas of the domain), and an arbitrary number (5) of latent variables (here we see that 5 is enough because it captures all the slow degrees of freedom), the model is able to learn the slow-variables values (eigenvalues controlling the evolution of the macroscopic variables) and then correctly reproduce the long-time behavior of the macroscopic observables (the X), and additionally produce realistic noise (x) / realistic microscopic fluctuations (this last part is not very clear, I am guessing here).
In addition, the latent representation of previously unseen initial conditions can be inferred (with decent error bars), allowing to predict the future of unseen initial trajectories.


Overall, the idea is interesting and supported by correct mathematical derivations and experimental proofs of concept.

Thus I lean on accepting the paper.


However, I have some questions and remarks that I would like to be addressed.


My main criticism would be the following: the paper does not use any pre-existing benchmarks, which makes accuracy comparisons essentially impossible.
Of course one may object that the novelty of the approach makes comparison impossible, however I think that it should be possible to re-implement State of the art methods (by the way I appreciated the literature review section), and show how much the new method deals well with smallish data sets (which I believe it does ?).

My second main criticism is that although the method is general and in principle applicable to complex problems, here it is applied to textbook cases, for which the exact macroscopic equations are known, and for which a mechanically stable state is reached in finite time, in other words, two very simple problems. (although if I recall well the equations (26), (27) do not allow a simple diagonal form in the style of (21)).
The simplicity of the problems attacked is seen also in the important (but hidden) remark, that equation (25) is sufficient to produce x, and that "no parameters need to be learned for p(x_t|X_t)".
What would happen for more complicated problems ? (e.g. protein folding, even in the case of small molecules like butane (C2H6 if I recall is a simple standard)? )
In particular, if the true number of "independent" components (number of lambda's) needed was very large ?


Also I have a trivial question, that however I think needs to be discussed in the paper. What is the error on the prediction of the microscopic positions of the {x} variables (the elementary particles) ? My guess is that by construction, the generative model has a 100% error (as good as random, I mean), because it does not track the future of individual particles, but rather predicts the future of the large-scale, slowly-evolving coarse-grained variables (X, governed by the latent z's).
Am I correct ? I think it may be worth mentioning that quickly, for the readers that are unfamiliar with the topic.
If I am completely wrong, it is then even more worth mentioning the accuracy of x(t+P) predictions.

I did not understand why section H of appendix was not part of the main text. It seems quite central to be able to forecast unseen initial conditions, and not just the future of already seen trajectories.




Less important remarks about how the paper presents the research:

sec 2.2 : additional intuitive explanation about how the learning is performed would be welcomed.
It is not clear to the unfamiliar reader as to how learning proceeds.
It seems to me that you directly learned the parameters of the joint probability distribution, which is quite factored, as you showed in Eq. (8), but needs Variational Inference methods to be solved for. What I don't see clearly is the shape of q_\phi, or rather, what it means for B to be bi-diagonal. Maybe you could explain which variables are connected with which, given your assumptions on B_\phi.
Also, after reading Appendix , Figure 9 and 11 could be compressed and included in the main text, or at least referred to explicitly.

A number of graphs are un-readable, in particular the x- and y-axis labels are so tiny that one cannot get what is plotted against what.
Please correct that, increasing both figure size and x/y-labels sizes.


-------
a couple of detailed remarks:

In Fig 13-16, the central plot does not have a colorbar. Is it because it's exactly the same as for the left plot ? If so, mention it or center the colorbar in a visually suggestive way

typo: "is is" (search it)


review_title: Good ideas - weak applications - several points to be improved

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a generative model for learning a low-dimensional representation of a dynamical system from high-dimensional observations. The novelty of the approach is to introduce two latent spaces, one representing the standard physics-agnostic latent space learned from the data and one representing physics-motivated variables. The goal is to learn the dynamics of the first layer, denoted by z_t, which is a coarse-grained representation of the dynamical system and mostly captures the slow processes that drive the system. 

Quality, clarity, originality and significance:
Pro: The paper is very well written, clear, and the generative approach is novel. Adding domain knowledge is relevant and significant when dealing with real world applications. Cons: Below are a few comments and questions to clarify some of the aspects and choices made. 

How are the physics-motivated variables chosen? Is there a guarantee that the chosen representation/variables are sufficient to allow a one-on-one mapping between the physics-based latent space $X_t$ and the observations/data $x_t$? (As a small comment, I found it slightly confusing to use the same letter for the latent space and data space, even if one is lower case and the other upper case.) 

What does equilibrium represent here? Does it necessarily need to be a stationary process as in the experiments, or can it be a trend, or a periodic signal? Are the latter two cases handled well by the method?

Fig. 1: Would it be possible to add the maps F and G to the figure? It is not very clear to me how the latent variables X are obtained, are they computed from the data? If X is the generative process for x, why do we need the latent space z? Would it be possible that the latent space X is sufficient, and if not, would the authors have examples when this is not the case?

The framework is applied to physics simulations. How easy/difficult would it be to apply the method to real world data, and what would be the main challenges?

Sect. 2: Should it be $x_t \in \mathcal{M}$, instead of $x \in \mathcal{M}$? The paper mentions that the $z_t$’s correspond to a nonlinear coordinate transformation, but do not specify of what?

What is the motivation for using the Ornstein-Uhlenbeck process for z, and what would be other alternatives? Why exactly does z need to be complex, is it in order to be able to capture slow and fast processes? In Figs. 2 and 5, the slow processes seem to have imaginary part close to zero, and fast processes real part close to zero. What is the explanation for this (this might be a known fact in dynamical systems theory, but not so much in ML)? Does it always have to be the case that the \lambda’s have either the real or imaginary parts close to zero? What would happen if the system is still multi-scale but on a much more continuous scale than in the experiments presented here and where there wouldn’t be such a clear distinction between slow/intermediate/fast scales (this would probably be the case in complex real world systems)? 

According to eq (4) to ensure long-term stability, the real part of $\lambda_j$ should be
negative. This only applies to the slow processes, is that right? What is an explanation for this? 

In the definition for $\sigma_j$ there is a 2 factor which is not in eq (7). Same line: the latent dynamics are stationary refers to which layer, $z_t$ or $X_t$? Using clearly defined names for the two latent spaces would make the paper easier to read. Maybe even add this to Fig. 1.

In ML we don’t see often complex distribution, and it would be good to specify that $\mathcal{CN}$ is the complex distribution.

Eq (1): as $\lambda$ and $\sigma$ do not depend on time, it means that all hidden processes $z_t$ follow the same dynamical model. What are the implications of this, and would this be a reasonable choice in a real world scenario?

How robust are the results to the choice of the number of latent variables h, and the choice of the physics-based variables?

--------------------------
Rebuttal: Thank you to the authors for their detailed response. I am happy with the response and will keep the original score.
review_title: Interesting paper that adds a physics-motivated latent space to the standard latent space

Meta-review: The consensus of the reviews is to accept the paper. I agree.

Reviewers highlighted many strengths, including a compelling main idea:
* R5: "The paper presents an interesting and motivating case for Bayesian inference in probabilistic generative models: a problem that has inherent uncertainty along with the ability to incorporate domain knowledge that can reduce the inference complexity."
* R3: "Overall, the idea is interesting and supported by correct mathematical derivations and experimental proofs of concept."
* R4: "the generative approach is novel. Adding domain knowledge is relevant and significant when dealing with real world applications"

As well as compelling experiments, substantially improved in the discussion period:
* R1: "The authors have shown some promising results in modeling particle dynamics."
* R5: "The addition of Appendix H, in my opinion, considerably strengthens the paper's story and case for acceptance. [... T]he authors have addressed most of my major concerns."

And clear writing:
* R5: "In general, the paper is well written (apart from some higher-level structural issues discussed below) and the notation is clear and unambiguous."
* R4: "The paper is very well written, clear"

The main weaknesses highlighted were in experiments (lacking good baselines, as well as ablations), and in discussing some choices in the model's construction. These were effectively addressed in the discussion (though R5 still points to some places that could be improved).
Decision: Accept (Poster)