Title: On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution
Venue: NeurIPS 2024
Accepted: yes
Keywords: hybrid modeling; identifiability; meta-learning
Abstract: The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of _hybrid deep generative models (hybrid-DGMs)_  that integrates known physics-based mathematical expressions in neural generative models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has not yet been theoretically probed or established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to consutrct a hybrid-DGM with theoretically-proven identifiability? This paper provides the first theoretical probe into the identifiability of hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs. On synthetic and real-data benchmarks, we provide strong empirical evidence for the un-identifiability of existing hybrid-DGMs using unconditional priors, and strong identifiability results of the presented meta-formulations of hybrid-DGMs.

On the Identifiability of Hybrid Deep Generative

Abstract

The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of hybrid deep generative models (hybrid-DGMs) that integrates known physics-based mathematical expressions in neural generative models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has not yet been theoretically probed or established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to consutrct a hybrid-DGM with theoretically-proven identifiability? This paper provides the first theoretical probe into the identifiability of hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs. On synthetic and real-data benchmarks, we provide strong empirical evidence for the un-identifiability of existing hybrid-DGMs using unconditional priors, and strong identifiability results of the presented meta-formulations of hybrid-DGMs.

1 Introduction

There has been increasing interest in integrating mathematical expressions of known physics with neural-network functions for hybrid, or gray-box, modeling [1, 2, 3, 4, 5]. The incorporation of physics-based inductive bias has the potential to improve the generalizability and interpretability of deep learning [4], while the expressive and flexible neural functions offer the opportunity to fill the gap in our prior knowledge of physics (or expert models ) [4, 5]. Recent progress in particular has started to see principled developments of hybrid deep generative models (DGM) where the data-generation process is described by a hybridization of physics-based and neural components. Both of these two components are then identified from data via, for instance, amortized variational inference [2, 4]. These hybrid-DGMs have shown benefits of intepretability, generalizability, and out-of-distribution (OoD) robustness, especially when learning to model physics systems from observational data [2, 4]. An important premise of these successes is that the hybrid expression of the physical systems can be accurately identified from the observed data: for example, if we can identify the parameters

and/or physics governing observed pendulum movements, we could then use the identified generative models/parameters for predicting the future trajectory of the pendulum [2].

However, substantial research based on the theory of nonlinear independent component analysis (nonlinear ICA) have established that a naive DGM or latent variable model is un-identifiable [6]. Active research has also reported various ways to construct an identifiable latent variable model, primarily via constructing a conditionally independent generative model [7, 8, 9, 6]. Most of the progress has been established for the identifiability of latent processes underlying time-series data, leveraging the temporal dependency of the latent variables [8] or observed domain index for nonstationary segments [7] to construct the conditional generative model. For static latent variables, the construction of conditionally independent generative model has mainly relied on the introduction of auxiliary observations such as class or domain labels [9].

How does the hybrid expression of a DGM impact its (un)-identifiability? Furthermore, since hybrid-DGMs are often intended to model physics systems with unknown parameters to be identified from data, what may be an effective approach to construct a hybrid-DGM with theoretically-proven identifiability? Unfortunately, despite the increasing interest in hybrid-DGMs, their identifiability remains unexplored. This identifiability however is critical for the intended use of a hybrid DGM, e.g., to predict beyond the observations used to identify the model, or to be robust in out-of-distribution (OOD) generalizations.

Recent efforts in hybrid-DGMs indeed have increasingly noted the challenges in properly learning the two components within the hybrid models, especially in ensuring a non-trivial solution to the physics-based component due to the presence of a highly-expressive neural component [1, 2]. Various strategies have been presented, such as regularizing the expressiveness of the neural component [2], trajectory-based and adaptive optimization to allow automatic complexity adjustment for the neural component [1], and expert-augmentation leveraging the physics-based component to improve generalization to distributional shifts [4]. No existing works, however, have probed or established the identifiability of hybrid-DGMs, theoretically or empirically.

In this work, we present the first theoretical probe of the identifiability of hybrid-DGMs. Importantly, forgoing the need of observed auxiliary variable to condition the generative model, we present meta-learning as a novel solution to establish the identifiability of hybrid-DGMs. To this end, we present a learn-to-identify formulation for meta-learning hybrid-DGMs, and theoretically establish the identifiability of these meta-hybrid-DGMs via conditional independence of the DGM given few-shot context samples. We further provide empirical identifiability results of these meta-hybrid-DGMs in a variety of synthetic and real datasets generated from physics systems, in contrast to existing non-meta hybrid-DGMs where identifiability cannot be established without observed auxiliary variables. Finally, we note that while this work focuses on hybrid-DGMs, the presented meta-learning formulation provides a general new condition to construct identifiable DGMs beyond the hybrid setting.

2 Related Works

Hybrid-DGMs: A number of hybrid models have emerged in various domains to combine physicsbased functions with neural networks, to compensate for unknown components in the known physics. While earlier approaches [10, 11, 12] focused on using neural networks to learn the residual between the measurements and those obtained from physics-based simulation, tighter integrations have been presented by, e.g., NeuralSim [13] and universal differential equations (UDEs) [14], to include neural networks as different components within a physics-based function. Most of these earlier works are focused on hybrid modeling in specific application domains.

More recently, several hybrid-DGMs have been presented [2, 1, 4, 3], with a focus on properly learning the physics component within the hybrid-DGM so it is not overpowered by the expressive neural component. In physics-integrated hybrid VAE [2], this was achieved by a regularized learning method that controls the expressiveness of the neural component while preserving the semantics of the physics-based component. Similarly, regularization of the expressiveness of the neural component is considered in [1]. In [4], expert augmentation is proposed to improve the OOD robustness of these hybrid-DGMs by fine-tuning on synthetic OOD samples, generated by sampling outside the training distribution of the physics-based component within the trained hybrid model.

Despite these efforts in better learning the two separate components within hybrid-DGMs, the identifiability of hybrid-DGMs has not been theoretically probed nor established. Empirically, no existing works have evaluated these hybrid-DGMs from the lens of identifiability metrics. Theoretically, no existing works have examined the conditions for constructing identifiable hybrid-DGMs. This work will take the first step to bridge these gaps, establish the un-identifiabilty of naive unconditional hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs.

Nonlinear ICA and Identifiability of DGMs: With nonlinear ICA, strong identifiability results have been established for unsupervised latent variable models [7, 8, 9, 6, 15, 16, 17, 18]. Fundamentally, construction of identifiable generative models is achieved by defining independence structure when conditioned on observed auxiliary variables [9]. Much progress has been made in identifying latent dynamic processes generating observed time series, including leveraging conditional independence of the DGM given time segment index temporal dependence [8], sparse temporal encoding [15], and even unobserved inferred domain labels for non-stationary time series [16, 17, 18]. For static latent variables, identifiability is mainly established via the introduction of auxiliary observations such as class or domain labels [9]. In specific, iV AE [6] established the identifiability of DGMs assuming an exponential families conditional distributions given observed auxiliary variables.

The reliance on observed conditioning variable, however, limits the applicability of these identifiability results to hybrid-DGMs: as hybrid-DGMs are intended for modeling physics system with unknown and often continuous parameters, it is not clear what observed auxiliary label may be available to construct an identifiable DGM. This work will forego this assumption of observed auxiliary variable and present meta-learning as a novel condition to construct identifiable DGMs. While presented for hybrid-DGMs, this connection between meta-learning and the identfiability of DGMs is a novel contribution to general non-hybrid DGMs that has not been constructed in the existing literature.

3 Problem Formulation: Unidentifiablity of Hybrid-DGMs

Let x ∈ R n be the observed random variables and z ∈ R d be the latent variables generating x , with the following generation process:

where p ( z ) = ∏ d i =1 p ( z i ) represents the prior distribution of independent generative factors z i , p ( x | z ) is the likelihood function which can be defined based on the mixing function x = F ( z ) .

Construction of Hybrid-DGMs: When constructing a generative model for Equation (1), if we assume a known mathematical expression explaining the mechanism of the mixing function as f P, we will have a physics-based expression of this generation process, with unknown parameters of f P as physics-based latent variables z P. If we have no prior knowledge, we can describe the mixing function with a neural network f N with abstract representation of the latent variables as z N - the latter is the foundation of many successful DGMs including variational autoencoders (VAEs) [19].

When understanding complex systems in many domains, recent years have seen an increasing interest in the middle-ground of the above two scenarios: some prior knowledge in the form of a mathematical expression f P of physics exists about the data observed, yet often inexact with unknown gaps to the actual data-generating mechanisms. This motivated recent developments of hybrid-DGMs where the mixing function x = F ( z ) is described as a combination of a physics-based and neural component, f P and f N, respectively, with corresponding physics-based and abstract latent variables z P and z N.

As a concrete example, suppose that we observe the time-series of the angular position φ of a damped forced pendulum system, x = [ φ 0 , φ 1 , . . . , φ T ] where φ i = φ ( i ∆ t ) , governed by:

where the first two terms describe the physics of an ideal pendulum system, the third term the damping effect, and the last term the external force. Parameters ω, ξ, A, and ϕ specify different systems governed by the same physics, representing the independent latent generative factors.

Suppose that we choose to leverage our prior knowledge about the ideal pendulum physics f P ( x ; z P ) = ¨ φ + z 2 P sin φ with unknown parameter ω . To bridge its potential gap with actual

data-generating physics, we then choose to complete it with a neural-network expression of an ODE f N θ ( x ; z N ) , parameterized by θ and with an abstract latent variable z N: while abstract, we hope f N θ to absorb the effect of the last two terms in the data-generating physics (Equation (2) with z N representing its remaining data-generating parameters ξ, A, and ϕ . This gives rise to a hybrid mixing function x = F [ f P , f N θ ; z P , z N ] = ODEsolve [ f P ( x ; z P ) + f N θ ( x ; z N ) = 0] .

Assuming Gaussian observation noises, we obtain the likelihood of the Hybrid-DGMs as:

with prior distribution for the latent variables assumed to be Gaussian as:

where the mean µ P and variance σ 2 P for p ( z P ) can be defined by prior knowledge due to its physical meanings. Note that z P will be directly interpretable and physically meaningful as they will be semantically grounded to the parameters of the physics model f P; z N, in comparison, will be abstract but need to absorb effect of the varying parameters underlying the missing physics f N.

The data-generating process of the hybrid-DGM with parameter θ can thus be defined as:

While described in the context of a conrete example, Equations (3)-(5) represent a general expression of hybrid-DGMs where the hybrid-mixing function x = F [ f P , f N θ ; z P ; z N ] can be designed in various forms (and not limited to the hybrid-ODE described in the example above).

Unidentifiability of Hybrid-DGMs: Now assume we have access to data D = { x (1) , . . . , x ( N ) } generated by p θ ∗ ( x , z P , z N ) , where θ ∗ is the true but unknown data-generating parameter. For an identifiable DGM, our goal is to learn θ such that the likelihood p ∗ θ ( x | z P , z N ) , the priors p ( z P ) and p ( z N ) , and the posteriors p ∗ θ ( z P | x ) and p ∗ θ ( z N | x ) are all correctly recovered.

In practice, because we only observe x without access to the latent variables z P or z N, we optimize θ to match the marginal density of x , e.g., via amortized variational inference to maximize the evidence lower bound (ELBO) of p θ ( x ) in VAEs, such that:

Unfortunately, even if the above optimization is done perfectly ( i.e., p θ ( x ) = p θ ∗ ( x ) ), there is no guarantee that p ∗ θ ( x | z P , z N ) , p ∗ θ ( z N | x ) , or p θ ∗ ( x , z P , z N ) are correctly identified. As shown in [6], for these densities to be recovered based on matching p θ ( x ) , the DGM needs to satisfy:

In another word, the matching of p ∗ θ ( x ) needs to imply a unique solution of θ ∗ to be obtained. In reality, however, it has been shown in [6] that DGMs with unconditional prior are unidentifiable. We refer the readers to the important theorems and the proof in [6, 9, 20] for more details, but to what extent is the hybrid-DGM affected by this theory of identifiability?

At a concept level, let us first consider the neural component ( z N and f N θ ) within the hybrid-DGM. For z N of any distributions, there are always nonlinear transformations that change its values but not its distributions [6]. In a highly expressive neural network, this transformation can be learned in the likelihood p θ ( x | z N ) . This will give the same fit of p θ ( x ) , but non-unique solutions of p θ ( x | z N ) and p θ ( z N | x ) , resulting in non-identifiable neural component within the hybrid-DGM.

For the physics-based component ( z P and f P θ ), this theory does not hold because the pre-specified form of f p can not trivially accommodate any transformations applied to z P without affecting the fitting of observations. In practice, however, because both components jointly contribute to p θ ( x | z P , z N ) as defined in Equation (3), the optimization of z P and f P θ maybe overpowered by the overly-expressive neural component, rendering trivial solutions to z P and f P θ .

In other words, identification of hybrid-DGMs is inflicted by two fundamental challenges: 1) the theoretical un-identifiability of its neural component, and 2) this un-identifiable but highly expressive component overpowering the identification of the otherwise identifiable physics-based component. While existing works [2, 1, 3] have focused on the second challenge, we address the problem from its fundamentals at the un-identifiability of z N and f N θ : we introduce a novel condition to construct identifiable hybrid-DGMs, and establish their identifiability both theoretically and empirially.

4 Learn-to-Identify Hybrid-DGMs via Meta-Learning

Based on nonlinear ICA theory [9], an identifiable DGM can be constructed by conditionally independent latent variables p ( z | u ) : in the literature, the conditioning variable u is often assumed to be additionally-observed side information such as class labels or domain indexes [6]. Here, we show that meta-learning formulations offer an opportunity to construct an identifiable DGM, with u taking forms of k -shot context samples sharing the same data-generation process as the query samples.

Construction of Identifiable Meta-Hybrid-DGMs: Given is a set of k context samples, D s , sharing the same data-generation process as query samples x , we define the generation of x as:

where ϕ = { θ, ζ } , p θ ( x | z P , z N ) is defined in Equation (3), and the conditional prior p ζ ( z N |D s ) is assumed to be factorized exponential family distributions [21]:

where Q i is the base measure, Z i ( D s ) is the normalizing constant, T i = ( T i, 1 , . . . , T i,e ) are the sufficient statistics that are fixed (not estimated). λ ζ i ( D s ) = ( λ ζ i, 1 ( D s ) , . . . , λ ζ i,e ( D s )) are the corresponding parameters conditioning on D s . In this paper, we realize this conditioning through a composite of a neural network h ζ ( x s ) , parameterized by ζ , for individual context samples x s ∈ D s , and an averaging function across all context samples:

Note that exponential families have universal approximation capabilities, thus this commonly-used assumption is not very restrictive [20].

Amortized Variational Inference: To enable inference over the hybird-DGM in Equation (8), we approximate the posterior density p ( z P | x ) as q η ( z P | x ) and p ζ ( z N | x , D s ) as q ζ ( z N | x ∪D s ) , the latter realized with the same network as defined in (9) but with the additional input x in addition to D s .

Formally, we cast the variational inference into a meta-learning formulation. Consider a dataset D with M similar but distinct data-generation process: D = {D m } M m =1 . For each D m , we consider disjoint context samples D s m = { x s, 1 , x s, 2 , . . . , x s,k } and query samples D q m = { x q, 1 , x q, 2 , . . . , x q,l } , where k ≪ l . Instead of maximizing the marginal likelihood of x for all x ∈ D , we formulate a metaobjective to learn to maximize the marginal likelihood p ( x q |D s m ) of all query samples x q ∈ D q m when conditioned on support set D s m , for all underlying data-generation processes m ∈ { 1 , 2 , . . . , M } :

where the first term represents likelihood of the meta-hybrid-DGM on query sample x q ∈ D q m , and the last two terms represent Kullback-Leibler divergences between the prior and posterior densities of z P and z N, respectively, where densities of z N are conditioned on context samples D s m . The maximization of Equation (11) is performed over { η, ζ, θ } in an episodic training where, in each episode, the division of D s m and D q m is shuffled within each D m . The likelihood is calcuated by the reparameterization trick [19], while the two KL-terms are calculated analytically.

Note that in Equation (11), the meta-hybrid-DGM is formulated on the neural component within the hybrid-DGM to establish its identifiablity. While the identifiability of the physics-based component is not hinged on this construction, in practice, it is possible to extend the conditional prior (Equation

(9)) to z P to obtain p ζ ( z P , z N |D s m ) , leading to an alternative meta-formulation:

We will empirically compare the two formulations above, and demonstrate how the identifiability of z P and z N is each affected by the proposed meta-construction of the hybrid-DGMs.

5 Identifiablility Theory for Meta-Hybrid-DGMs

We consider the meta-formulation a novel condition to ensure the identifiability of unsupervised DGMs. Here, we show that, built on nonlinear ICA and the theory of identifiability for conditionalVAEs established in [6], the presented meta-hybrid-DGMs are identifiable.

For readability, below we collect the formulations of the hybrid-DGM from previous sections:

where ϕ = ( θ, T , λ ζ ) are model parameters and λ ζ is defined in Equation (10). Equation (14) gives a more general definition than Equation (3) where ε is an independent noise variable.

Definition 1: Let ∼ be an equivalence relation on the parameter space Φ . We say the DGM in 1 is ∼ -identifiable if p ϕ ( x ) = p ˜ ϕ ( x ) ⇒ ϕ ∼ ˜ ϕ

Definition 2: Let ∼ be the equivalence relation on Φ defined as follows:

where A is a de × de matrix and c is a vector of dimension de , d being the dimension of z N and e the dimensionality of the sufficient statistics T i . If A is invertible, we denote this relation by ∼ A .

Definition 2 establishes a specific equivalence relation that allows to recover the sufficient statistics of the generative model up to a linear matrix multiplication.

Theorem 1: Assume we observe data sampled from p θ ( x , z P , z N |D s ) as defined in Equations (13) -(15) with parameters ϕ = ( θ, T , λ ζ ) . Assume the following holds:

(i) The set { x ∈ X| φ ε ( x ) = 0 } has measure zero, where φ ε is the characteristic function of the density p ε defined in p ε ( x -F [ f P , f N θ ; z P , z N ]) .

(ii) The hybrid mixing function F θ is injective.

(iii) The sufficient function T i,j are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero.

(iv) There exist de +1 distinct context sets D s, 0 , D s, 1 , . . . , D s,de such that the de × de matrix L defined as follows is invertible:

Then the parameters ϕ = ( θ, T , λ ζ ) are ∼ A -identifiable.

The proof for Theorem 1 directly builds on that presented in [6]. Condition (iv) in Theorem 1 establishes that, if the generative factors in the neural component of the hybrid-DGM is d -dimensional each specified with e -dimensional sufficient statistics, we will need observations from a minimum of de +1 distinct and independent generation processes in order to identify the hybrid-DGM. Considering the pendulum-system example discussed in Section 3 and Equation (2), since the neural component is generated by three independent system parameters ξ, A, and ϕ ( d = 3 ) and assuming Gaussian statistics ( e = 2 ), the hybrid-DGMs will be identifiable if we have observations generated from more than 3 × 2 + 1 = 7 distinct combination of parameter values for ω, ξ, A, and ϕ in the true data-generation process (Equation (2)). We will empirically verify this condition in Section 6.

6 Experiments and Results

Data: We considered three simulated and one real-world benchmarks for hybrid-DGMs, including three simulated physics systems of forced damped pendulum [2], advection-diffusion system [2] and double pendulum [4], and one real-world system double pendulum [4]. For each of the three simulated physics systems, we randomly sampled the initial states and parameters of the governing function to generate observations. We refer to the governing functions used for the generation of data as full physics functions, and design partial physics functions for represent our imperfect prior knowledge about the observed data, reflecting a variety of potentially additive and multiplicative errors - these partial physics functions are used as the f P in the hybrid models. In data generation, we varied the parameters in the components both present and absent in the prior physics. More details on each dataset can be found in Appendix A.

Baselines: We considered 1) physics-integrated hybrid V AE [2] and 2) APHYNITY [1], both using regularization to control the expressiveness of the neural component. Note that while the original formulation of APHYNITY does not have an inference network to identify z P or z N (making it intended for data generated from a single parameter setting), we added an inference network (that shares a similar architecture with the other two models) for fairness of comparison.

Metrics: With a focus on identifiability, we focus on the following metrics: 1) for physics-based latent variable z P, we consider mean squared error (MSE) owing to its physical meaning; 2) for abstract latent variable z N modeled in the neural component, we consider the mean correlation coefficient (MCC) between the true data-generating parameters and z N sampled from its learned posterior density. The calculation of MCC follows standard practice in the identifiability literature [6, 9]: to evaluate ∼ A -identifiablity, we calculate the weak MCC with the method described in [22], where a high MCC provides evidence of successful identification. We also evaluate a more strong identifiable relation ∼ P with strong MCC and the details can be found in Appendix E.

In addition to metrics on z P and z N, we further introduce MSE metrics on the generated x in two distinct scenarios: 1) reconstruction MSE measures how well an observed x is reconstructed from the identified z P and z N, and 2) prediction MSE measures how well the identified z P and z N can be used to generate outside the observed x used to identify them, e.g., for predicting over long time domains, or predicting for a different sample that comes from the same generation process but with different initial conditions. We use these two metrics to showcase that a good reconstruction does not guarantee a good identification, while the ability to use the identified latent variables to predict for different samples (other than observed) may be a surrogate for identifiaiblity.

Ground Truth

APHYNITY

Hybrid-VAE

Meta-Hybrid-VAE

coS9

Reconstruction

Generation

Reconstruction

Generation

1.5

1.0

1.00

1.0

0.95

1.0

0.5

0.90

0.9

0.5

0.0

0.85

0.8

0.0

0.80

-0.5

0.5

0.75

0.7

1.0

0.70

-1.0

0.65

0.6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

Time(s)

Time(s)

Time(s)

Time(s)

ForcedDampedPendulum

DoublePendulum

6.1 Results on Synthetic Data

Reconstruction vs . Prediction Performance: Table 1 summarizes quantitative identifiability metrics on the presented meta-hybrid-VAE in comparison to its baselines, where the prediction performance of all models is mentioned by their ability - once identified - to predict for a sample sharing the same data-generating parameters but different initial conditions. As shown, meta-hybridVAE significantly improved the identifiability of the abstract latent variable z N (the highest MCC close to 1) and via which, moderately improved the accuracy of the physics-based latent variable z P (the lowest MSE). Very importantly, while both baseline models identified z P with a reasonable accuracy, their MCC values were significantly lower - providing strong empirical support for the un-identifiability of the neural component of the hybrid-DGM as theorized in this paper.

Also importantly, all hybrid models achieved comparable reconstruction MSE for observed x 's, where the meta-hybrid-VAE did not necessarily produce the lowest MSE. The ability of the identified hybrid-DGM to predict for different samples other than observed, however, varied significantly: while the meta-hybrid-VAE was able to deliver a MSE comparable to the reconstruction task, the two baseline models saw a deterioration of MSE by two magnitudes. Fig. 1 provides visual examples to demonstrate this performance difference between reconsruction and prediction tasks, stressing the importance to 1) look beyond reconstruction performance for evaluating hybrid-DGMs, and 2) consider prediction of unobserved samples as a potential surrogate for identifiability measures when ground-truth of latent variables is not available (real data settings in Section 6.2).

Interestingly, among the three datasets, the advection-diffusion system appeared to be relatively simple to identify as the baselines did not exhibit as significant a degradation of performance in the more difficult identifiability metrics (MCC and prediction-MSE). This is potentially because the missing component from the prior physics has a small effect. In comparison, the MCC and prediction-MSE of the baseline models were significantly poorer in double pendulum, suggesting a difficult missing physics to be identified. Regardless, meta-hybrid-V AE was able to obtain significant margins of improvements compared to baselines across all datasets.

Predictions over Longer Time Domains: Fig. 2 shows the prediction performance of the three models when predicting the trajectory of pendulum movement beyond the time domain used to identify the model, with a visual sample of forced damped pendulum system. As shown, while all models' performance deteriorated as they predicted outside the time domain used in training, the presented meta-hybrid-VAE demonstrated significantly smaller drop in performance and slower rate of deterioration as the prediction horizon increased. This provides strong evidence regarding the importance of the identifiability, even for the neural component, of a hybrid model for the purpose of predictive tasks.

OODPerformance: Figure 3 (left) summarizes the performance of the three models in settings where either the physics-based component z P or the neural component z N was outside the training distribution for forced damped pendulum system. Results demonstrated that, meta-hybrid-VAE obtained the strongest performance in OOD settings for both the identification of the physics and neural components. Details of the OOD parameters can be found in the Appendix A.1.

Empirical Verification of the Condition for Identifiability: Theorem 1-iv) stipulates that the identifiability of the hybrid-DGM depends on the number of independent conditional densities observed, in relation to the dimension of the latent factors to be identified. To experimentally

Ground Truth

APHYNITY

Hybrid-VAE

Meta-Hybrid-VAE

Ground Truth

0.12

Hybrid-VAE

Meta-Hybrid-VAE

ANAHV

0.10

2.0

MSE

0.08

Angle

1.5

90°0

1.0

+0°0

2

0.02

0.5

-1

2.5

3.0

3.5

4.0

4.5

5.0

0.00 4

1.0

1.2

1.4

1.6

1.8

2.0

0.0

2.0

2.4

2.8

3.2

3.6

4.0

0

2

3

TimeDomain

TimeDomain

TimeDomain

Time

ForcedDampedPendulum

Advection-DiffusionSystem

DoublePendulum

VisualizationExample

APHYNITY

Hybrid-VAE

Meta-Hybrid-VAE

MSE of z,↓

MCC↑

1.0

0.9

0.20

0.8

0.8

0.15

0.6

0.7

0.10

0.4

0.6

0.05

0.2

0.5

0.00

ID

OODzp

0OD1N

0.0

ID

OODZp

OODZN

The number of tasks

10

12

14

16

verify this, we evaluated the identifiability metric MCC of the hybrid-DGM when learning over data generated from an increasing number of distinct parameter values governing the pendulum system (Equation (2)), i.e., number of tasks in meta-learning. The results in Figure 3 (right) showed that the MCC metric exceeded 0.85 when 7 tasks appeared in the training data and stablized when 8 or more tasks appeared. This agreed with the theorem that, for the pendulum system that had a 3-dimensional parameter vector to be identified in the neural component of the hybrid model with a Gaussian assumption of 2-dimensional sufficient statistics, a minimum of 3*2+1 = 7 distinct 'tasks' s needed to identify the system.

Additional Alation Analyses of Meta-Hybrid-VAE: In Appendix C.1, we provided ablation results on the pendulum system to demonstrate that the identifiability results of the presented metahybrid-VAE were minimally affected by the number of parameters to be identified, as long as the theoretical condition for identifiability is met. In Appendix C.2, we provided further ablation studies to show that the physics-based component did not suffer from the type of un-identifiabilty discussed in this paper as the neural component, although the meta-formulation did moderately improve the accuracy of the estimation of the physics-based parameters.

6.2 Real Data of Double Pendulum

Experimental settings: We used the dataset of a double pendulum introduced by [4], which contains 21 videos of the pendulum. Each run lasts approximately 40 seconds and is recorded at 400Hz. We extractd the position of the pendulum limbs from each frame with elementary computervision tools. We divided each video into observations of x 's to consist of 20 temporal frames with a sampling frequency of 100 Hz. Because there is no clear indication of which samples belong to the same data-generation process, we use 7 samples preceding the current query sample as the context samples. We generated 10000, 3400, and 3400 training, validation, and test samples, respectively.

We adopted Equation (19) as the physics-component of the hybrid-DGM, using the known lengths of the two arms ( L 1 = 91 mm and L 2 = 70 mm ) and assuming ˜ m = 1 . The total energy of the double

GroundTruth

APHYNITY

Hybrid-VAE

Meta-Hybrid-VAE

coS9

cOS9

1.0

0.35

1.0

1.0

0.40

0.9

0.8

0.8

0.45

0.8

0.6

0.6

0.50

0.55

0.7

0.6

0.4

0.4

0.60

0.5

0.2

0.65

0.4

0.0

0.2

0.70

0.3

0.2

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175

0.2

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175

Time(s)

Time(s)

Time(s)

Time(s)

Reconstruction

Prediction

pendulum decreases over time in all videos, which is not explained by this physics model. Nor does the physics model consider potential vibrations or errors in extracting the arms' positions, which is expected to be learned by the neural component of the hybrid-DGM with the dimension of z N = 4.

Results: As ground-truth generative factors are not available in the real data, we relied on prediction MSE as a surrogate for the identifiability metric as informed by results in Sections A.1-A.3. While all models delivered comparable reconstruction MSE, meta-hybrid-VAE was able to achieve a prediction MSE at a level similar to reconstruction while the two baselines experienced a deterioration by 2-3 magnitudes. Visual examples are provided in Fig. 4. This strong performance in real data provides solid evidence for the presented meta-formulation to establish identifiability of hybrid-DGMs.

7 Conclusions & Discussion

In this paper, we probe the un-identifiability of hybrid-DGMs with unconditioned priors, and present meta-learning as a novel solution to construct identifiabile hybrid-DGMs - both results were supported by strong theoretical and empirical evidence. Moving forward, this work can be improved as follows.

Comprehensive ablation studies of the meta-hybrid-VAE are needed to examine the effect of several key hyperparameters, such as the size of k -shot context set and the dimension of z N. The latter was set to the number of true generative factors in synthetic experiments, and its effect on model performance needs to be further examined. Note that this setting was identical across baselines, thus it had no effect on the significant margins of improvements seen by meta-hybrid-VAE compared to baselines. Future ablation studies should also delve into the benefit of hybrid-DGMs in comparison to purely physics-based and purely neural DGMs, which we considered out of scope of the current study that focused on establishing the identifiability of hybrid-DGMs both theoretically and empirically.

Regarding the generality of the meta-formulation as a solution to identifiable DGMs beyond the hybrid formulation, Appendix D presented an initial investigation on the performance of the presented meta-formulation on a general non-hybrid VAE on a synthetic dataset of non-stationary Gaussian timeseries, in comparison to the identifiable-V AE [6] constructed using known class labels to condition the generative model.

Future works could investigate this effect in a broader context. Finally, while the meta-formulation foregoes explicitly observed auxiliary variable, it does assume the ability to pair context and query samples from the same data generation process. This is a common assumption in meta-learning and, in scenarios where this knowledge is not evident, simply pairing by preceding temporal samples was shown to be effective in our real-data experiments. In scenarios where no information is available for pairing, the presented meta-solution will not be applicable.

Acknowledgments and Disclosure of Funding

This study is supported by the National Key Research and Development Program of China(No: 2020AAA0109502), the Talent Program of Zhejiang Province (No: 2021R51004), the NIH NHLBI grant R01HL145590, the NIH NINR grant R01NR01830, and NSF grant OAC-2212548.

References

Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124012, 2021.

Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems , 34:14809-14821, 2021.

Ziming Liu, Yuanqi Du, Yunyue Chen, and Max Tegmark. Physics-augmented learning: A new paradigm beyond physics-informed learning. In NeurIPS 2021 AI for Science Workshop , 2021.

Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, and JörnHenrik Jacobsen. Robust hybrid learning with expert augmentation. Transactions on Machine Learning Research , 2023.

Zhaozhi Qian, W. R. Zame, L. M. Fleuren, Elbers P, and M. van der Schaar. Integrating expert ODEs into neural ODEs: Pharmacology and disease progression. In NeurIPS , 2021.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems , 29, 2016.

Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics , pages 460-469. PMLR, 2017.

Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics , 4(26):eaau5872, 2019.

Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics , 36(4):1307-1319, 2020.

Florian Golemo, Adrien Ali Taiga, Aaron Courville, and Pierre-Yves Oudeyer. Sim-to-real transfer with neural-augmented robot simulation. In Conference on Robot Learning , pages 817-828. PMLR, 2018.

Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 9474-9481. IEEE, 2021.

Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385 , 2020.

David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In ICLR , 2021.

Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. In NeurIPS , 2023.

Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. In NeurIPS , 2021.

H Halva and A Hyvarinen. Hidden markov nolinear ica: Unsupervised learning from nonstationary time series. In UAI , 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyv, Revant Kumar, et al. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research , 18(57):1-59, 2017.

Christopher M Biship. Pattern recognition and machine learning (information science and statistics). Springer New York , 2007.

Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems , 33:12768-12778, 2020.

A Details on Experimental Data Generation

A.1 Forced Damped Pendulum

We generated data from the full physics function in Equation (2), with observed x = [ φ 0 , . . . , φ i , . . . , φ T ] , φ i = φ ( i ∆ t ) , T = 49 , ∆ t = 0 . 05 . We randomly sampled the initial condition φ 0 ∈ ( -1 . 57 , 1 . 57] (with ˙ φ 0 = 0 fixed). Physics parameters [ ω, ξ, A, ϕ ] to be identified were varied as follows: ω ∈ [0 . 785 , 3 . 14] , ξ ∈ [0 . 0 , 0 . 8] , A ∈ [0 , 40] and ϕ ∈ [3 . 14 , 6 . 28] . We generated total 60000 samples and separated them into a training, validation, and test sets with 4,0000, 10000, and 1,0000 samples, respectively. In all hybrid-DGMs, we considered the first two terms n in Equation (2) as the partial physics , thus z P = ω and z N with a dimension of 3 is expected to recover [ ξ, A, ϕ ] . This setting induces an additive error in the prior physics. In OOD setting, we set OOD z P as ω ∈ [3 . 14 , 3 . 5] and OOD z N as ξ ∈ [0 . 8 , 1 . 0] , while keeping the rest of the parameters unchanged.

A.2 Advection-diffusion System

We generate data from the following full physics function:

where s is the spatial dimension. The solution T ( s, t ) was observed as x := [ T 0 , . . . , T T ] , where T j := [ T (0 , t j ) , . . . , T ( s max , t j )] T at t j := j ∆ t is uniformly distributed in space with a dimension of 20. We set T = 49 , ∆ t = 0 . 02 , the boundary condition as T (0 , t ) = T ( s max , t ) = 0 , and the initial condition as T ( s, 0) = c sin( πs/s max ) with c ∈ [0 . 5 , 1 . 5] . Physics parameters to be identified [ a, b ] were sampled as: a ∈ [0 . 01 , 0 . 1] and b ∈ [0 . 01 , 0 . 1] . We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMs, we set partial physics as the first two terms of Equation (18), thus z p = a and z N with a dimension of 1 is expected to recover b . This represents an additive error in the prior physics.

A.3 Double Pendulum

We generated data from the following full physics function:

where ˜ m = m 1 /m 2 . The observed x := [( φ 1 , 0 , φ 2 , 0 ) , . . . , ( φ 1 ,j , φ 2 ,j ) . . . , ( φ 1 ,T , φ 2 ,T )] , where ( φ 1 ,j , φ 2 ,j ) is the value of a solution at t j := j ∆ t with T = 79 , ∆ t = 0 . 025 , L 1 = L 2 = 1 . 0 . We randomly sampled the initial condition φ 1 , 0 , φ 2 , 0 ∈ ( -1 . 57 , 1 . 57] (with ˙ φ 1 , 0 = ˙ φ 2 , 0 = 0 ). Physical parameters [ G, ˜ m ] to be identified were sampled as follows: G ∈ [5 . 0 , 15 . 0] and ˜ m ∈ [0 . 5 , 1 . 5] . We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMs, we set partial physics to be Equation (19) with ˜ m = 1 , thus z p = G and z N with a dimension of 1 is expected to recover ˜ m . This induces a multiplicative error in the prior physics.

B The proof of Theorem 1

Theorem 1: Assume we observe data sampled from p θ ( x , z P , z N |D s ) as defined in Equations (13) -(15) with parameters ϕ = ( θ, T , λ ζ ) . Assume the following holds:

(i) The set { x ∈ X| φ ε ( x ) = 0 } has measure zero, where φ ε is the characteristic function of the density p ε defined in p ε ( x -F θ [ f P , f N θ ; z P , z N ]) .

(ii) The hybrid mixing function F θ is injective.

(iii) The sufficient function T i,j are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero.

(iv) There exist de +1 distinct context sets D s 0 , D s 1 , . . . , D s de such that the de × de matrix L defined as follows is invertible:

Then the parameters ϕ = ( θ, T , λ ζ ) are ∼ A -identifiable.

Proof: This proof mainly refers to [6]. For simplicity, we denote F θ as f , F ˜ θ as ˜ f , λ ζ as λ and λ ˜ ζ as ˜ λ . Suppose we have two sets of parameters ( f , T , λ ) and ( ˜ f , ˜ T , ˜ λ ) such that p f , T ,λ ( x |D s ) = p ˜ f , ˜ T , ˜ λ ( x |D s ) for all pairs ( x , D s ) . Then:

By taking the logarithm on both sides of equation (28) and replacing p T ,λ by its expression from (9), we get:

Let D s 0 , D s 1 , . . . , D s de be the context sets provided by assumption (iv) of the Theorem, and define ¯ λ ( D s l ) = λ ( D s l ) -λ ( D s 0 ) . We plug each of those D s l in (29) to obtain de +1 such equations. We subtract the first equation for D s 0 from the remaining de equations to get for l = 1 , . . . , de :

Let L be the matrix defined in assumption (iv), and ˜ L similarly defined for ˜ λ . Define b l = ∑ i log ˜ Z i ( D s 0 ) Z i ( D s l ) Z i ( D s 0 ) ˜ Z i ( D s l ) and b the vector of all b l for l = 1 , . . . , de . Experssing equation (30) for all points u l in matrix form, we get:

We multiply both sides by the transpose of the inverse of L T from the left to find:

where A = L -T ˜ L and c = L -T b .

Now by the definition of T and according to assumption (iii), its Jacobian exists and is an de × d matrix of rank d . This implies that the Jacobian of ˜ T ◦ ˜ f -1 exists and is of rank d and so is A . We distinguish two cases:

(1) If e = 1 , then this means that A is invertible (becuase A is d × d ).

(2) If e > 1 , define ¯ x = f -1 ( x ) and T i (¯ x i ) = ( T i, 1 (¯ x i ) , . . . , T i,e (¯ x i )) . According to Lemma 3 in [6], for each i ∈ [1 , . . . , d ] there exist e points ¯ x 1 i , . . . , ¯ x e i such that ( T ′ i (¯ x 1 i ) , . . . , T ′ i (¯ x e i )) are linearly independent. Collect those points into e vectors (¯ x 1 , . . . , ¯ x e ) , and concatenate the e Jacobians J T (¯ x l )

evaluated at each of those vectors horizontally into the matrix Q = ( J T (¯ x 1 ) , . . . , J T (¯ x e )) (and similarly define ˜ Q as the concatenation of the Jacobians of ˜ T ( f -1 ( ˜ f (¯ x ))) evaluated at those points). Then the matrix Q is invertible (through a combination of Lemma 3 and the fact that each component of ˜ T is univariate). By differentiating equation (32) for each x l , we get (in matrix form): Q = A ˜ Q . The invertibility of Q implies the invertibility of A and ˜ Q .

Hence, equation (32) and the invertibility of A mean that ( f , T , λ ) ∼ A ( ˜ f , ˜ T , ˜ λ )

C Alation Analyses of Meta-Hybrid-VAE

C.1 Effect of the Dimension of Unknown Parameters

Here, we provided ablation results on the pendulum system to demonstrate that the identifiability results of the presented meta-hybrid-V AE were minimally affected by the number of parameters to be identified, as long as the theoretical condition for identifiability is met. The results can be seen in Table 2.

C.2 Meta-component on cp

Here, we provided further ablation studies to show that the physics-based component did not suffer from the type of un-identifiabilty discussed in this paper as the neural component, although the metaformulation did moderately improve the accuracy of the estimation of the physics-based parameters. The results can be seen in Table 3.

D Results on General Non-Hybrid VAEs

We compare our meta-formulation of the non-hybrid VAE (Meta-VAE) compared to the original VAE and identifiable V AE (iV AE) on a synthetic datasets about non-stationary Gaussian time-series in [6]. To evaluate the performance of the method, we compute the MCC between the original sources and the corresponding latent samples from the learned posterior. The results can be seen in Table 4.

E Strong MCC results

We also evaluate proposed model on a more strong identifiable relation ∼ P defined as follow:

Definition 3: Let ∼ be the equivalence relation on Φ defined the same as 16:

ZNI

ZN3

3.0

2.5

2.0

1.5

-1

1.0

2

1.0

1.52.02.53.0

-1

Hybrid-VAE

3.0

2.5

2.0

1.5

-2

1.0

0.5

0.5

1.01.52.0

2.5

3.0

3

Meta-Hybrid-VAE

If A is a block permutation matrix, we denote this relation by ∼ P .

The theory according to ∼ P is following:

Theorem 2: ( e ≥ 2 ) Assume the hypotheses of Theorem 1 hold and that e ≥ 2 . Further assume:

(2.i) The sufficient statistics T i,j in Equation 15 are twice differentiable.

(2.ii) The mixing function F θ has all second order cross derivatives.

then the parameters ( θ, T , λ ζ ) are ∼ P -identifiable.

Theorem 3: ( e = 1) Assume the hypotheses of Theorem 1 hold and that e = 1 . Further assume:

(3.i) The sufficient statistics T i, 1 not monotonic.

(3.ii) All partial derivatives of F θ are continuous.

then the parameters ( θ, T , λ ζ ) are ∼ P -identifiable.

The proof of Theorem 2 and 3 can be found in [6]. Then to evaluate the performance on ∼ P , we calculate the strong MCC between learned latent variables and ground truth variables (Strong MCC Ground Truth) and strong MCC between runs of the model (Strong MCC Different Runs) [6, 9, 22]. The results can be found in Table 5 and we provide a visualization example of forced damped pendulum system to show the relation of learned latent variables between two different runs.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our method and experiments have show our main claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed it on Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs of the theorem in the main paper can be found in Appendix B .

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided enough information to reproduce the results in Section 6 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The data and code will be released soon.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide experimental settings and details on Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the standard deviation of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer:[NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper thoroughly assesses the issue of identifiability of both the neural and physics components in hybrid deep generative models (hybrid-DGMs), and proposes a novel approach to formulate such models using meta-learning. The authors show the performance of the model in comparison to other hybrid-DGMs and the ground truth for three well-known physics examples.

STRENGTHS:
The paper is the first theoretical study of the identifiability of hybrid-DGMs.

WEAKNESSES:
Possible limitations of applicability related to specifics of meta-learning.

QUESTIONS:
How resource heavy this method is compared to the other methods used in the comparison?
How well does it perform for OOD testing in terms of robustness and generalisation?
How well it does predict the behaviour of dynamical systems over longer time intervals (is there a performance degradation over time)? 
Would meta learning using the Wasserstein distance instead of the KL divergence improve the identification of both the physics and the neural components?
As suggested by the authors themselves this work would benefit from in-depth hyperparameters to performance study, particularly taking into account the relationship between data samples and model performance.

Comments:
- as a minor point - there are some references that didn't compile as well as typos in the manuscript.
- another minor point is that the appendix is missing with the default NeurIPS instructions left in place - this can be removed.

LIMITATIONS:
The authors didn't provide a separate limitations section but discussed the limitations of their work and future directions in the Discussion/Conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the identifiability of Hybrid Deep Generative Models and proposes a Meta-Learning-based approach to address the (un)identifiability issue of current DGMs. Theoretical discussions are provided, and experiments are conducted to verify the proposed method.\n\nSTRENGTHS:\nThis paper adopts the result from [1] gives a discussion on the indetifiability issue of Hybrid Deep Generative Models's data driven part.\n\n\n[1] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.\n\nWEAKNESSES:\n1. In \"Augmenting Physical Models with Deep Networks,\" identifying the physical part and the data-driven part is important, as mentioned in previous work such as [1]. As mentioned by the authors, this paper focuses on the identifiability issue of the latent variables and the model parameters of the data-driven model. However, the motivation for identifying this part is unclear since it does not have any physical interpretation. The main goal of the data-driven part is more about approximation accuracy rather than interpretability (identifiability). Therefore, the reviewer is not fully convinced by the motivation.\n\n2. The authors also agree that there are two sides to the identifiability issue: (1) physical parameters and parameters from the data-driven part, and (2) the identifiability of the parameters in the data-driven part. This paper focuses on the latter. However, assuming there is no identifiability issue with (1), the problem degrades to the same setting as [2]. Therefore, the theoretical discussion seems borrowed from paper [2], without any revisions.\n\n3. The writing needs to be further improved, including the definition of notations.\n\n[1] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124012, 2021.\n\n[2] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.\n\nQUESTIONS:\n1. In Definition 2, what is $T(\\cdot)$? Is that the sufficient statistic? Please define it explicitly.\n\n2. In Theorem 1, in assumption (ii), what is $\\mathcal{F}_\\theta$? What is the difference between $\\mathcal{F}_\\theta$ and $\\mathcal{F}(f_P, f_{N_\\theta}; Z_P, Z_N)$?\n\nLIMITATIONS:\nThe major limitation is the motivation of this work, as the reviewer mentioned in the weakness section. Another weakness is the lack of technical contribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA method and a theory of learning hybrid deep generative models (esp., VAEs) are proposed. The method is based on meta-learning, and the theory is about the identifiability of the neural component's latent variable. The method is also empirically compared to baseline methods of hybrid DGMs.\n\nSTRENGTHS:\n- In the introduction, the motivation for analyzing the identifiability of hybrid DGMs is clearly stated.\n- The potential connection to the study around nonlinear ICAs sounds reasonable and interesting.\n- The method is technically reasonable. To my knowledge, meta-learning has not clearly been applied in the context of hybrid modeling.\n- Experiments are done with reasonable, yet mostly synthetic, datasets and with adequate baseline methods.\n\nWEAKNESSES:\n(1)\nit is unclear what is the theoretical contribution particularly by this work. The theory in Section 5 looks like a mere rehash of what was presented in [6]. Also, the theory in Section 5 focuses only on the neural network part of hybrid DGMs, and the presence of the physics part of the model does not seem to affect the discussion. This limits the significance of the theoretical contribution.\n\nMoreover, as the authors pointed out, the un-identifiability of hybrid DGMs stems both from 1) the un-identifiability of the neural component and 2) the over-powering effect of the neural component. The authors say that they focus on 1), but then the discussion becomes simply about non-hybrid, general DGMs, which makes the authors' claim that they theoretically analyzed the identifiability of *hybrid* DGMs questionable.\n\nMinor points:\n- In Eq. (13), the prior of $z_P$ is missing.\n- What makes the synthetic and real double pendulum data different? E.g., is there supposed to be any source of non-negligible noises?\n\nQUESTIONS:\nPlease elaborate on the point (1) in the Weaknesses section above.\n\nLIMITATIONS:\nLimitations nicely discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The original contribution for his manuscript is a metalearning framework to identify the physical parameters in a hybrid deep generative models.

STRENGTHS:
This article starts with a good question about the identifiability of physical parameters in hybrid deep generative models (that include physics as an inductive bias). Solving this question could answer key questions in data-driven knowledge discovery. This article presents  meta learning framework which could help identify the physical parameters statistically.

WEAKNESSES:
This article seems to be a work in progress. The text alone is not an end product. Symptomatically of this work in progress, the central concept of this article of identifiability is misspelled more than 10 times throughout. The contributions of this paper are not clearly stated and hard to find. It seems that the theory stated in this article was already published elsewhere, as illustrated by the reference to another paper for the proof of the only theorem of this article.

QUESTIONS:
The proof-of-concept results on pendulum models and the reaction-diffusion equation have very few parameters to identify and do not inform that the method would be useful in more realistic or more complex models. Could you provide examples with more parameters to identify? Could you give theoretical support to how fast your method would converged to the identified quantities of interest as the number of parameters to identify increases?

LIMITATIONS:
Limitations are discussed in the text.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15