Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling
Authors: Naoya Takeishi
Alexandros Kalousis
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models
variational autoencoders
physics-integrated machine learning
gray-box modeling
hybrid modeling

Abstract: Integrating physics models within machine learning models holds considerable promise toward learning robust models with improved interpretability and abilities to extrapolate. In this work, we focus on the integration of incomplete physics models into deep generative models. In particular, we introduce an architecture of variational autoencoders (VAEs) in which a part of the latent space is grounded by physics. A key technical challenge is to strike a balance between the incomplete physics and trainable components such as neural networks for ensuring that the physics part is used in a meaningful manner. To this end, we propose a regularized learning method that controls the effect of the trainable components and preserves the semantics of the physics-based latent variables as intended. We not only demonstrate generative performance improvements over a set of synthetic and real-world datasets, but we also show that we learn robust models that can consistently extrapolate beyond the training distribution in a meaningful manner. Moreover, we show that we can control the generative process in an interpretable manner.


Reviews:
[{"review_id": "gLCKG5Yil26", "reviewer": "Reviewer_c9CR", "summary": "This paper describes a VAE architecture in which some latent variables are semantically related to our prior knowledge of the physical model. For instance, this represents the (unknown) angular velocity or diffusion coefficient. The semantic is defined the\nfunctional form of  the decoder part. Then the objective function is augmented with regularization terms to better control how the model will take into account the physical model and the prior knowledge. The experimental part relies on two synthetic datasets and two reals ones. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The interaction between Data and theory-driven modeling is clearly an important topic.  The introduction addresses the topic in maybe a too broad perspective. The authors want to keep their work as general as possible, but you can think about a lot of possibilities depending on the task or the scientific area (model, optimization, generation, generalization, robustness, ... ). The paper could be easier to follow with an example of purpose. \n\nThe example provides in section 2.1 really helps to understand the purpose. Maybe it could appear in the introduction (you should avoid the confusing use of theta, NNet parameters for the decoder in 2.2.2, and part of the observation in 2.1 and 5.1). Then the real purpose becomes clearer: some latent variables are semantically related to our prior knowledge of the physical model. For instance, this represents the (unknown) angular velocity or diffusion coefficient. The semantic is defined the functional form of  the decoder. \n\nThe contributions of the paper are clearly interesting. However, there is a for me a clarity issue. The notations (with theta) are sometime misleading. The section 3 describes some nice ideas but it is really difficult to follow.  For instance the section \"on grounding physics encoder by physics-based data augmentation\"  describes a nice idea but would it be possible to simplify a bit the discourse to go a bit more straight to the point. \n\nFinally the experimental part (section 5) does not really provide a meaningful insight. Maybe, the galaxy part could be discarded to spare place for the others. Moreover, the results on advection-diffusion system are not so easy to interpret. They are a bit different from the pendulum one. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "cHPsf5jqStf", "reviewer": "Reviewer_EtH5", "summary": "This paper develops a general approach to integrating physical domain knowledge into the latent space, encoder, and decoder of VAEs. The latent space is separate into a physics component and learned component, where the former is grounded to some unobserved physical quantity or system variable. The decoder is a composition of a physics-based component with known functional form (e.g. an ODE of the latent physical variable) and a neural component which also takes the learned part of the latent space as input. The encoder is a composition of two stages, the first of which extracts the learned latents, and the second of which recovers the physically grounded latent variables. In order to train this model to encourage interpretability and usage of the physics components, the authors develop several regularization terms which are added to the ELBO loss, thereby constrain the power of highly flexible neural encoders and decoders, ensuring that the physical components play a meaningful role in explaining the data. The importance of such regularization (and the value of physics components) is demonstrated on a series of physical system identification tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a nice paper which generalizes existing approaches that integrate physical domain knowledge within partially specified ODEs (e.g. Yin et al, ICRL 2021), while allowing for the specification of priors over latent system variables, as in the VAE framework. To my knowledge, this paper is original and valuable in the following ways:\n1. It generalizes integration of physical domain knowledge into ML models beyond the additive combination of learned components with physics ODEs.\n2. It applies physics integration in the context of Bayesian generative modeling, allowing for a range of additional tasks like unsupervised learning, counterfactual simulation / sample generation, and Bayesian parameter estimation.\n2. It shows how to overcome the well-known problem of VAE decoders and latent spaces being overly flexible (and hence ignoring expert or physical guidance through priors and expert-specified equations) through regularization.\n\nThe experiments also appear to be sufficiently thorough in demonstrating both the value of integrating physical knowledge and the importance of regularization when learning the behavior of physical systems like diffusion-advection, galaxy image generation, and human gait prediction. For these reasons I recommend acceptance of the paper.\n\nI just have a few questions / points of clarification. Regarding the proposed regularization terms, I wonder if there are Bayesian interpretations for them, especially given the reference to posterior predictive checks? For example, would it be possible to derive something like $R_{PPC}$ by imposing a prior on the form of the decoder, similar to how the prior on the latent space acts as a regularizer in variational inference? \n\nI also wonder whether the $R_{PPC}$ method is going to work well in the case where the observations are a pretty complex / high dimensional function of the physical variables, e.g. images of humans walking from the joint velocities in the case of human gait. In that case, while the full decoder should be able to do reasonable job of matching the observed data, it seems like the physics-only decoder would struggle, unless the function $h_A$ were a learned complex function like a CNN as well.\n\nIn this sort of situation then, it seems like having the $R_{PPC}$ term would actually hurt learning, because the physics-only decoder would do a very bad job of fitting the data even after training / conditioning, and the full decoder would be regularized towards being similar. This could be fixed if modelers had enough domain knowledge to specify $h_A$ (e.g. $h_A$ could be a differentiable renderer in the case of generating images of humans walking), but it seems like this won't always be the case. It would be good if the authors could discuss this situation in the final manuscript.\n\nFinally, I was a bit confused by the use of the term \"harness\" to mean something like \"constrain\" or \"regularize\", because \"harness\" also has the meaning of \"utilize / exploit / make use of\". It might be a good idea to replace \"harness\" with \"regularize\" or \"constrain\" in phrases like \"harness trainable components\".\n\n== Post-Rebuttal ==\n\nThank you to the authors for their detailed responses and clarifications. I believe that they have adequately responded to the questions raised by myself and other reviewers, and I will maintain my score of 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Ideally the authors would discuss both the limitations and societal impact more -- I have highlighted some potential limitations in my review above. Regarding societal impact, maybe the authors could mention how physically grounded models would aid better scientific discovery, but also lead to more accurate / robust surveillance and prediction systems (e.g. in the case of identifying and predicting human gait).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "_FNf9hriME", "reviewer": "Reviewer_LEU9", "summary": "The authors propose to incorporate physics models into variational autoencoder (VAE) architectures in a principled manner. They design regularization terms in the ELBO loss to achieve a VAE with a latent space that is partially grounded in interpretable physics elements. The experiments demonstrate that the resulting VAE performs well in reconstruction error, is able to infer physically interpretable parameters in its latent space, and generalizes beyond the provided data.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There has recently been a lot of work in the space of neural networks grounded in physics. The authors provide a reasonable literature review in the main paper, and further contextualization in existing work in the appendix. Since I do not work directly in this space, it is difficult for me to judge the exact novelty of the proposed methods. That being said, the approach seems to be effective, and the authors do their due diligence in extensive experimentation and comparison to methods similar in motivation. I would recommend the authors further expand their literature review to include works in the space of physics-grounded learning for the purpose of control. I provide some examples below.\n\n[1] Gupta, Jayesh K., et al. \"Structured mechanical models for robot learning and control.\" Learning for Dynamics and Control. 2020.\n\n[2] Richards, S. M., et al. \"Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems.\" Robotics science and systems. 2021.\n\nThe ideas in the submission are evaluated extensively across multiple experimental domains. The experiments strongly support the claims made in the paper in terms of reconstruction quality, latent space interpretability, and the flexibility of the VAE to handle non-additive physics-based and learned components. The general structure of the baselines considered poses an ablation study where each of the ablations resemble a related work in the space. However, some of the baselines (e.g., phys-only) seem a bit contrived for a number of the experimental settings. For example, in the case of the pendulum and the advection-diffusion system, the phys-only baseline has no hope of performing well as the authors only provide it with partial information about the simulated physics. Granted, this is not the case for the galaxy and human gait experiments. While only qualitative results are shown for the physics-only model in the main paper, quantitative results are reported in the appendix for these experiments and the trends are consistent with the main paper. It would have been good to provide a bit more intuition for why and how the proposed model is able to outperform the baselines (for example in lines 277-281). \n\nOne question I had while the reading the paper is how does the VAE ensure that the learned $f_{A}$ component is not zero since all of the regularization terms appear to focus on driving the VAE closer to the physics model. Further experimental analysis demonstrating the performance of the learned $f_{A}$ term would be helpful.\n\nMy biggest concern with the paper is its lack of discussion of the limitations of the proposed method. The authors do refer to a general limitation in the space of physics-based learning in the difficulty of integrating complex physics models. However, no specific limitations of the proposed approach are discussed. Furthermore, there appears to be a lack of transparency in the discussion of the results for the advection-diffusion experiments. Table 1 reports that the proposed method underperforms the considered baselines in terms of reconstruction ability. This on its own is not grounds for rejection as there are extensive experiments demonstrating the proposed method's value. However, the discussion in the paper does not address this limitation, but rather ignores it (lines 309-310). The authors should discuss their hypothesis for why this drop in relative performance was observed for this experiment.\n\nI have a few additional clarification questions with respect to the experiments:\n\n1. Why were the results for the NN+phys (with no regularization) baseline not displayed in Fig. 1?\n\n2. In Fig. 2, were both the $z_{P}$ and the $\\omega$ values changed for each of the plots?\n\n3. For the pendulum experiments, why was the $f_{A,2}$ term introduced? What is the purpose of the latent variable $z_{A,2}$ beyond the function of $z_{A,1}$?\n\n4. In Table 1, why were standard deviations reported instead of standard error?\n\nAlthough the paper is very well written, the notation made it difficult to follow at times. I may have missed this definition earlier in the paper, but is $f$ defined prior to line 97 which is where it appears to be first mentioned? Line 106 was difficult to follow notationally. The space of $S_{f_{p}}$ is not specified. I would at first assume that $S_{f_{P}}$ is in $Z_{P}$, but in the second half of the statement, it serves as the sole argument of $f_{A}$. This is confusing since the domain of $f_{A}$ is set to $Y_{P} \\times Z_{A}$.\n\nThe most difficult section to read was Sec. 3.2, which is critical to the understanding of the paper. I had to read this section over multiple times and I am still not sure I am following the reasoning. First, I do not understand how $\\mathcal{F}$ is a functional but it is able to take a value output from $h_{A}$ instead of a function as an argument (line 181). What is the difference between $x^*$ and $x^*_{z^*_{P}}$? Lines 187-194 were particularly confusing and difficult to understand. There is a lot of recursive reasoning and notation here. For example, $x'$ should resemble $x^*$ which is the physics-only reconstruction from the original data $x$ and latent code $z_{A}$. In Eq. (9), the output of $g_{P,1}$ is made to be close to $x'$, which is in turn computed using $g_{P,1}$. In Eq. (10), the input to $g_{P,2}$ is $x^*_{z^*_{P}}$ instead of $x'$, which is meant to provide self-supervision for $x'$ to be close to $x^*_{z^*_{P}}$. Simplifying the notation and streamlining the logic in this section would help the reader understand the proposed method better.\n\nI found a couple minor typos in the paper (see below) but otherwise the polish of the paper was very good. \n\n1. Typo in line 71: 'One one' should read 'On one'.\n\n2. Line 207: 'Ones...are one' is awkward grammatically.\n\n3. Typo in line 95: 'solve' should be 'solves'.\n\n4. The references should be proofread (e.g., to ensure the year is not entered twice in a citation).\n\nOverall, the authors propose an interesting and empirically successful method for physics-based learning using VAEs. Their approach demonstrates good generalizability and encourages a partially interpretable latent space. With the additional of a transparent discussion of the limitations of the method and improved notational clarity, this work will be a good contribution!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned above, the authors have not adequately addressed the limitations of their specific method in the paper (besides general limitations with respect to incorporating complicated physics models into gray-box models). Discussion of the drop in reconstruction performance in Table 1 (advection-diffusion experiments) would be a good candidate for the discussion of limitations. Furthermore, I would encourage the authors to think about potential inadvertent societal consequences of the proposed method such as any concerns with deployment of neural network architectures in safety critical systems.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to integrate a physics-based model within the Variational Auto-Encoder framework. Two of the reviewers are in favor of the paper and point out that the paper possesses many interesting features:
- The paper generalizes the integration of physical domain knowledge into ML models beyond the additive combination of learned components with physics ODEs.
- It applies physics integration in the context of deep generative modeling.
- It shows how to overcome the well-known problem of VAE decoders and latent spaces being overly flexible through regularization.
- The ideas in the paper are evaluated extensively across multiple experimental domains.

The third reviewer is inclided towards the rejection of the paper. However, the reviewer did not take part in the discussion and spent only an hour on the paper.
Overall, I find the points raised by the reviewers properly addressed by the authors in the rebuttal. Similarly, two reviewers seem to be satisfied with the rebuttal. Therefore, I am in favor of accepting the paper.
Decision: Accept (Poster)