On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution
Authors: Yubo Ye
Maryam Toloubidokhti
Sumeet Vadhavkar
Xiajun Jiang
Huafeng Liu
Linwei Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: hybrid modeling; identifiability; meta-learning
Primary area: generative_models
TL;DR: We propose the learn-to-identify framework to improve the identifiability of hybrid-DGMs.

Abstract: The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of _hybrid deep generative models (hybrid-DGMs)_  that integrates known physics-based mathematical expressions in neural generative models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has not yet been theoretically probed or established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to consutrct a hybrid-DGM with theoretically-proven identifiability? This paper provides the first theoretical probe into the identifiability of hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs. On synthetic and real-data benchmarks, we provide strong empirical evidence for the un-identifiability of existing hybrid-DGMs using unconditional priors, and strong identifiability results of the presented meta-formulations of hybrid-DGMs.

On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution
Yubo Ye"
Maryam Toloubidokhti
Zhejiang University 222301310zju. edu. en
Rochester Institute of Technology
mt61290rit.edn
Sumeet Vadhavkar Rochester Institute of Technology sv62348rit.ed
Xiajun Jiang
Huafeng Liu
Rochester Institute of Technology xj 277566rit.edd
Zhejiang University Buuhf@zju edu cn
Linwei Wang
Rochester Institute of Technology linwei. wangârit. edu edu
Abstract
The interest in leveraging physics- inductive bias in deep learning has resulted in recent development of hybrid deep generative models (hybrid-DGMs)) that integrates known physics-basee mathematical expressions models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has yet been theoretically probed established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to consutret a Dybrid-DGM with theoretically-proot probe into the identifiability of hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs On synthetic and real-data benchmarks, we provide strong empirical evidence for the un- un-identifiability of existing hy Hybrid-DGMs using unconditional priors, and strong identifiability results of the presented formulations hybrid-DGMs.
neural generative
identifiability? This paper provides the first theoretical
Introduction
There has been increasing interest in integrating mathematical expressions of known physics with
neural-network functions for hybrid, gray-box, modeling 2 4 (5). The incorporation of physics-basee inductive bias has the potential to improve the generalizability and interpretability of deep learning 44. while the expressive and flexible neural functions offer the opportunity fill the gap in our prior knowledge of physics (or expert models) (5 Recent progress in particular has started see principled developments of hybrid deep generative models (DGM) where the data-generation process is described by hybridization of physics-basse and neural components, Both of these two components are then identified from data via, for instance, amortized variational inference 4 These hybrid-DGMs have shown benefits of intepretability, generalizability, and out-of-din of--offriibbutiinn (OoD) robustness, especially when learning to model physics systems from observational data 2 An important premise of these successes is that the hybrid expression of the phy systems can be accurately identified from the observed data: for example, if we can identify the parameters
Work was done visiting exchange student Rochester Institute of Technology.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and/or physics governing observed pendulum movements, we could then use the identified generative models/paramettrs for predicting the future trajectory of the pendulum (2].
However, substantial research based on the theory of nonlinear independent component analysis (nonlinear ICA) have established that naive DGM latent variable model is un-identifiable [6]. Active research has also reported various ways to construct an identifiable latent variable model, primarily via constructing conditionally independent generative model g 6 Most of the progress has been established for the identifiability of latent processes underlying time-series data, leveraging the temporal dependency of the latent variables [8] or observed domain index for non- stationary segments [7] to construct the conditional generative model. For static latent variables, the construction of conditionally independent generative model has mainly relied on the introduction of auxiliary observations such as class or domain labels (9].
How does the hybrid expression of a DGM impact its (un)-identifabbility? Furthermore, since hybrid-DGMs are often intended to model physics systems with unknown parameters to he identified from data, what may be an effective approach construct hybrid-DGM with theoretically-pront proven identifiabiiity? Unfortunately, despite the increasing interest in hybrid-DGMs, their identifiability remains unexplored. This identifiability however critical for the intended use of hybrid DGM, e.g., to predict beyond the observations used to identify the model, or to be robust in (OOD) generalizations.
Recent efforts in AbbridDDMM indeed have increasingly noted the challenges in properly learning the two components within the hybrid models, especially in ensuring non-trivial solution to the physics- based component due to the presence of highly- strategies have been presented, such as regularizing the expressiveness of the neural component 221. trajectory-based and adaptive optimization to allow automatic complexity adjustment for the neural component []], and sspert-augmentation leveraging the physies-based component to improve generalization to distributional shifts 4 No existing works, however, have probed or established the identifiability of hybrid-DGMs, theoretically or empirically.
neural component [[2]. Various
In this work, we present the first theoretical probe of the identifiability of Hyyrid-DGMM, Importantly, forgoing the need of observed auxiliary variable to condition the generative model, we present meta-learning as a novel solution to establish the identifiability of hybrid-DGMss. To this end, we present a learn-to-identify formulation for meta-learning hybrid-DGGMs, and theoretically establish the identifiability of these meta-hybrid-DDe context samples. We further provide empirical identifiability results of these meta-hybi meta-hybrid-DGMM in variety of synthetic and real datasets generated from physics systems, in contrast to existing non-meta hybrid-DGMs where identifiability cannot be established without observed auxiliary variables. Finally, we note that while this work focuses on hybrid-DGMs, the presented meta-learning formulation provides a general new condition to construct identifiable DGMs beyond the hybrid setting.
via conditional independence of the DGM given few-shot
2 Related Works
A number of hybrid models have emerged in various domains to combine physics- based functions with neural networks, to compensate for unknown components in the known physics. While earlier approaches [10, 1 [2] focused on using neural networks to learn the residual between the measurements and those obtained from physics-based simulation, tighter integrations have been presented by, e.g., NeuralSim IK and universal differential equations (UDEs) [14] to include neural networks as different components within a physics- based function. Most of these earlier works are focused on hybrid modeling in specific application domains.
Hybrid-DGMs:
More recently, several hybrid-DGMs have been presented learning the physics component within the hybrid-DGra DGM so it is not overpowered by the expressive neural component. In telysics-integaan hybrid VAE (2) this was achieved by regularized learning method that controls the expressiveness of the neural component while preserving the semantics of the physics-based component. Similarly, regularization of the expressiveness of the neural component is considered in D. In 4 expert augmentation is proposed to improve the OOD robustness of these hybrid-DGMs by fine-tuning on synthetic OOD samples, generated by sampling outside the training distribution of the physics- based component within the trained hybrid model.
with a focus on properly

---

Despite these efforts in better learning the two separate components within hybrid-DGMs, the identi- fiability of hybrid-DGMs has not been theoretically probed nor established. Empirically, no existing works have evaluated these hybrid-DGMss from the lens of identifiability metrics. Theoretically, no existing works have examined the conditions for constructing identifiable hybrid-DGMs. This work will take the first step to bridge these gaps, establish the un-identifiabilty of naive unconditional hybrid-DGMe and present meta-learning as novel solution to construct identifiable hybrid-DGee -s..s.
Nonlinear ICA and Identifiability of DGMs:
With nonlinear ICA, strong identifiability results have been established for unsupervised latent variable models 7 g 9 15 17 [8]. Fundamen-
tally, construction of identifiable generative models is achieved by defining independence structure when conditioned on observed auxiliary variables [9] Much progress has been made in identifying latent dynamic processes generating observed time series, including leveraging conditional indepen- dence of the DGM given time segment index temporal dependence R sparse temporal encoding [15), and even unobserved inferred domain labels for von-stationary time series [66 17 [8]. For static latent variables, identifiability is mainly established via the introduction of auxiliary observations such as class or domain labels [9]. In specific, iVAE [6] established the identifiability of DGMs assuming an exponential families conditional distributions given observed auxiliary variables.
The reliance on observed conditioning variable, however, limits the applicability of these identifiability results to MsM:: as hybrid-DGMs are intended for modeling physics system with unknown and often continuous parameters, it is not clear what observed auxiliary label may be available to construct an identifiable DGM. This work will forego this assumption of observed auxiliary variable and present meta-learning as a novel condition to construct identifiable DGMs. While presented for Sabrid-DGMs, this connection between meta-learning and the identfiability of DGMs is a novel contribution to general non-hybrid DGMs that has not been constructed in the existing literature.
3 Problem Formulation: Unidentifiablity of Hybrid-DGMs
Let X R be the observed random variables and Rdd be the latent variables generating x, with the following generation process:
p(x, p(())((x(z) p(x) -f p(xx z z)dz
(I)
where p(z) = I p(z) represents the prior distribution of independent generative factors zi. p(x(z) is the likelihood function which can be defined based on the mixing function x F(z).
Construction of Hybrid-DGMs:
When constructing a generative model for Equation ( if we assume known mathematical expression explaining the mechanism of the mixing function as fr. we will have a physics-based expression of this generation process, with unknown parameters of fp as physics- based latent variables Zp. If we have no prior knowledge, we can describe the mixing function with a neural network fN with abstract representation of the latent variables as the latter the foundation of many successful DGMs including variational autoencoders (VAEs) 19.
When understanding complex systems in many domains, recent years have seen an increasing interest in the middle-ground of the above two scenarios: some prior knowledge in the form of a mathematical expression fr of physics exists about the data observed, yet often inexact with unknown gaps to the actual data-generating mechanisms. This motivated recent developments of hybrid-DGMs where the mixing function x = F(z) (z) is described as a combination of a physics-basee and neural component, fr and SN. respectively, with corresponding physics-basd based and abstract latent variables Zp and zn.
As a concrete example, suppose that we observe the time-series of the angular position oo of a damped forced pendulum system, = [40, (i.....,........................ ,PT] where pp, p(iAt), governed by:
d' + w2 sin sp()) dip(t) A cos di2
(2)
dt
where the first two terms describe the physics of an ideal pendulum system, the third term the damping effect, and the last term the external force. Parameters w. A, and ó specify different systems governed by the same physics, representing the independent latent generative factors.
Suppose that we choose to leverage our prior knowledge about the ideal pendulum physics fr(x; zp) + + 2 sin ip with unknown parameter To bridge its potential gap with actual
3

---

data-generating physics, we then choose to complete it with neural-networ expression of an ODE IN, (x; zx). parameterized by & and with an abstract latent variable 2x: while abstract, we hope IN, to absorb the effect of the last two terms in the data-generating physies (Equation (2) with en representing its remaining data-generating parameters ç, A, and Φ. This gives rise to a hybrid mixing function x = F[fp, J' Zp, 2x| ODEsolve [fp zp) + JNN. (x; NN) = 0).
Assuming Gaussian observation noises, we obtain the likelihood of the Hybrid-DGMs as:
p [x Zp, 2x) = (x JN, 2p, zN], Ex) with prior distribution for the latent variables assumed to be Gaussian
(3)
p(zp)= == (zplpp.
p(zx) := (0)
(4)
where the mean μ4p and variance of for p(zp) can be defined by prior knowledge due to its physical meanings. Note that Zp will be directly interpretable and physically meaningful as they will be semantically grounded to the parameters of the physics model fe: AN. in comparison, will be abstract but need to absorb effect of the varying parameters underlying the missing physics fN.
The data-generating process of the hybrid-DGM with parameter can thus be defined as:
pe(x) = ]
pe (x, Zp, ZN) pe (x|ze,
po Zp, ZN Zx)dZZZZN
(5)
While described in the context of a conrete example, Equations 3 represent general expression
of hhybhdd-DGii where the hybrid mixiiii function x = Fl... IN.: Zp; zn] can be designed in various forms (and not limited to the hybrid-ODE described in the example above).
Unidentifiability of Hybrid-DGMs: Now assume we have access to data D D= (x(")..... x x(>}
generated by pg- (x, 2p; 2x), where 0 the true but unknown cata-generating parameter. For an identifiable DGM, our goal is to learn @ such that the likelihood p (x|2p, ZN), the priors p(zp) and p(zx), and the posteriors p (Zp|x) and pa (ZN x) are all correctly recovered.
In practice because we only observe x without access to the latent variables or Zx. we optimize 0 to match the marginal density of x, e.g., amortized variational inference to maximize the evidence lower bound (ELBO) of (x(x) in VAEs, such that:
(x(x) pg (x)
(6)
Unfortunately, even if the above optimization is done perfectly (i.e., po = po- (x) ), there is no guarantee that (x Zp, (N), (Z (x), or Pa- (x, up, Zy) are correctly identified. As shown in [6], for these densities to be recovered based on matching px)),, the DGM needs to satisfy:
((0,00 0) : pa = (x) +
(7)
In another word, the matching of p p needs to imply a unique solution of 0* to be obtained. In reality, however, it has been shown in 66 that DGMs with unconditional prior are unidentifiable. We
refer the readers to the important theorems and the proof in 66 9 20 for more details, but to what extent is the hybrid-DGM affected by this theory of identifiability?
At a concept level, let us first consider the neural component (2y and fN,) within the hybrid-DGM. For ZN of any distributions, there always nonlinear transformations that change its values but not its distributions 66. In a highly expressive neural network, this transformation can be learned in the likelihood Pa (x xx). This will give the same fit of (x),, but non- non-unique solutions of po (x (x(zx) and po po(2x|x)) x), resulting in non-identifiable neural component within the hybrid-DGM.
For the physics- based component (zp and fr,). this theory does not hold because the pre-specified form of fo can not trivially accommodate any transformations applied to Zp without affecting the fitting of observations. In practice, however, because both components jointly contribute to po(x|zp, /p. zn) as defined in Equation (3). the optimization of Zp and fp, maybe overpowered by the overly-expressive neural component, rendering trivial solutions to Zp and fr.
In other words, identification of hybrid-DGMS is inflicted by two fundamental challenges: 1) the theoretical un un-identifiabiliiyy of its neural component, and 2) this un-identifiable but highly expressive component overpowering the identification of the otherwise identifiable physics-based component. While existing works 2 i 3 have focused on the second challenge, we address the problem from its fundamentals at the un-identifiability of an and IN,: we introduce a novel condition to construct identifiable hybrid- Daardd and establish their identifiability both theoretically and empirially.

---

4 Learn-to-Identify Hybrid-DGMs via Meta-Learning
Based on nonlinear ICA theory [9], an identifiable DGM can be constructed by conditionally independent latent variables p(z|u): in the literature, the conditioning variable is often assumed to be additionally-obserrved side information such as class labels or domaín indexes [6]. Here, we show that meta-learning formulations offer opportunity to construct an identifiable DGM, with taking forms of k-shot context samples sharing the same data-generation process as the query samples.
Construction of Identifiable Meta-Hybrid-Dee sharing the same data-generation process as query samples x, we define the generation of x as:
Given set of k context samples, D",
/
P zp, 2n (D*)= = po (x|2x
(xx| (D"), Pe (x(x")
Po Po 2p, 2N D dezpda
(8)
where e {0,5]. Pv (x kpp ZN) defined in Equation 3 and the conditional prior Pe (ZN (D*) is
assumed to be factorized exponential family distributions (21:
PPAM = II Qe(ZN,s) ==1 (D())) exp (T,, (D*)
Pc(ZN(D*)= = (xx
where Qi is the base measure, Zi(D") is the normalizing constant, T; = (Tr,,..., Ti,e) are the sufficient statistics that are fixed (not estimated). (D (A (D")...., (D.)))) are the cor- responding parameters conditioning composite of neural network hx(xx), parameterized by C. for individual context samples x E D=, and an averaging function across all context samples:
D In this paper, we realize this conditioning through
Ac (D.) = D
Σ hc(x*) x'E
(10)
Note that exponential families have universal approximation capabilities, thus this commonly-used assumption is not very restrictive [20].
Amortized Variational Inference:
To enable inference over the hybird DGM in Equation (
approximate the posterior density p(ze|x) as q, (Zp]xx x) and Pc (ZN x, D*) as qe (zn x U UD). ). the latter realized with the same network as defined in (9) but with the additional input in addition to D=.
Formally, we cast the variational inference into a meta- meta-learning formulation. Consider a dataset D with M similar but distinct data data-generationn process: D = {Dr, MMM For each Dm- we consider disjoint context samples D; = {x* xx. x * } and query samples Dt m = {x*.1 .x.2...., x*1), where & << Instead of maximizing the marginal likelihood of x for all x E D. we formulate meta- objective to learn to maximize the marginal likelihood p of all query samples x" E Dr when conditioned on support set D for all underlying data-gened generatieern processes E (1.2...., M):
M
M
log log
N
{E (et)( Po [zp, 2x)
m= X
m= X D%
(Z)(g)zzp()"))lptta xx
(L( X U Dr Dit.) (D:t))
(11)
where the first term represents likelihood of the Seitaahhbrrd-DGGi the last two terms represent Kullback-Leiller divergences between the prior and posterior densities of Zp and Zy. respectively, where densities of Zn are conditioned maximization of Equation (TT) is performed over {n. Ç. 0) in an episodic training where, in each episode, the division of D and DV, is shuffled within each D, The likelihood is calcuated by the reparameterization trick [19] while the two KL-terms are calculated analytically.
query sample x E Dr in and
context samples Dr The
Note that in Equation 11 the meta-hybrid-DGGM is formulated on the neural component within the hybrid-DGM establish its identifiablity. While the identifiability of the physies-based component not hinged on this construction, in practice, is possible to extend the conditional prior (Equation
5

---

p
to Zp to obtain (Zp, ZN Din, leading to an alternative meta- meta-foomuuation:
M
log
{e (Z. (lo po Po
zN)l
(12)
m= X
m== X
[LL(q(Zp, 2N/x² U Dip zN (D..))
We will empirically compare the formulations above, and demonstrate how the identifiability of Zp and zy is each affected by the proposed meta-construction the hybrid-DGMs.
Identifiablility Theory for Meta-Hybrid-DGGMs
We consider the meta-formulation a novel condition to ensure the identifiability of unsupervised DGMs. Here, we show that, built nonlinear ICA and the theory of identifiability for conditional- VAEs established in [6], the presented meta-hybrid -GMed-rMrrd- are identifiable.
For readability, below collect the formulations of the hybrid-DGNi DGM from previous sections: (x( 2p, EN (D*) po Zz, (x))p(ZP)PPT.... (zN (D*)
(13)
(x Zp, 2N) Ps x Flfe, SN.i Zp, zx])
(14)
Q (=N,i)
PTA |N D*)
II 1-1
Z, (D*) exp (Ex.))) (D.)
(15)
where o (0, T, Aç) are model parameters and X is defined Equation (TO). Equation (14 gives more general definition than Equation (3) where is an independent noise variable.
Let be an equivalence relation the parameter space 4 We say the DGM in is ~-identifiable if pf() Px(x) = ó
Definition 1:
Definition 2: Let ~ be the equivalence relation on 4 defined as follows:
(,,T,><) (0, Tx)) c 'x)x'(x)) AT(F, (x)) vx
(16)
where A is de de matrix and is vector of dimension de, d being the dimension of zy and the dimensionality of the sufficient statistics T,, If A is invertible, we denote this relation by ~A
Definition 2 establishes specific equivalence relation that allows to recover the sufficient statistics of the generative model up to linear matrix multiplication.
Theorem Assume we observe data sampled from pa zp, Zn) as defined in Equations
13
15) with parameters (0, T, Aç).). Assume the following holds:
(1) The set {x E (x) 0} has measure zero, where Pe is the characteristic function of the density Pa defined (x Flfp, J'N; Zp, zx]).
(ii) The hybrid mixing function Fe is injective.
(ii) The sufficient function Te,; are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero.
(iv) There exist de distinct context sets Da.0 Da. "e......,Da.... De such that the de x de matrix defined as follows is invertible:
= (A(DD=)))> - (((D:))-\(D*0))
-
(17)
Then the parameters o- = (0, T,T) are ~ ~ ~A-identifiable
The proof for Theorem directly builds on that presented in 16. Condition (iv) in Theorem I establishes that, if the generative factors in the neural component of the hybrid-DGM d-dimensional each specified with e-dimensional sufficient statistics, we will need observations from a minimum of de+1 distinct and independent generation processes in order to identify the hybrid-DGM. Considering the pendulum-system example discussed in Section 3 and Equation 2 since the neural component generated by three independent system parameters &, and á (d = 3) and assuming Gaussian statistics (e 2), the hybrid-DGMs will be identifiable if we have observations generated from more than - distinct combination of parameter values for w, E, A, and á in the true data-generation process (Equation ( We will empirically verify this condition in Section

---

Table 1: Quantitative identifiability metrics for the presented meta-hybrid- VAE in comparison to physics-integrated hybrid- VAE [2] and APHYNITY M1, including MSE and MCC metrics on the latent variables, well as MSE of generated x during reconstruction of observed samples (Rec) and prediction of unobserved samples (Pre).
Forced Damped Pendulum
Advection Diffiionn-Dfffiion System
APHYNITY
Hybrid- VAE
Meta-Hybrid-VAr
APHYNITY
HyyrridVAEE Meta-Hybrid-VAk
MSE of zr 1
6.96(0.01)e-2
4.14(0.29)e-2
1.910.07)e-2
1.9(0.3)e-2
7.12(5.32)o-4
1.4(0.75)e-4
MCC
0.79(0.02)
0.59(0.03)
0.99(0.00)
0.98(0.01)
0.94(0.00)
0.99(0.00)
MSE x. (Rec)
5.2(0.01)e-2
2.66(0.03)e-2
2.91(0.06)e22
6.22(0.41))e-3
4.90(0.75)e-3
2.93(0.22)e-3
MSE of (Pre) |
2.37(0.67)
1.74(0.20)
6.85(1.11)e-2
8.25(0.8
8.53(572))-2
3.63(1,34)--3
Double Pendulum
Real Double Pendulum
APHYNITY
Meta-Hybrid-VEE Hybrid-VAE
APHYNITY
Hybrid-VAE
Meta-Hybrid HAE
MSE of 2r
4.58(0.04)c-1
3.97
3.8510.19)e-1
MCC 1
0.50(0.00)
0.51(0.00)
0.98(0.00)
MSE x (Rec)
4.88(0.00)e-2
4.05(0.25)e-2
3.8310.06je-2
3.78(0.52)e-2
2.20(0.14)e-3
2.67(0.34)o-2
MSE of (Pre)
1.29(0.23)ee
5.52(0.06)
2.8010.48)e-1
5.16(0.69)
1.87(..1))
2.88(0.34)e-2
Experiments and Results
Data: We considered three simulated and one real-world benchmarks for hybrid-DGMs, including three simulated physics systems of forced damped pendulum (2). advection-diffusion system 2 and double pendulum [4) and one real-world system double pendulum 44. For each of the three simulated physies systems, we randomly sampled the initial states and parameters of the governing function to generate observations. We refer to the governing functions used for the generation of data as full physics functions, and design partial physics functions for represent our imperfect prior knowledge about the observed data, reflecting a variety of potentially additive and multiplicative errors- these partial physics functions are used the fr in the hybrid models. In data generation, varied the parameters in the components both present and absent in the prior physics. More details on each dataset can be found in Appendix A
Baselines: We considered 1) physics-integrated hybrid VAE 2 and 2) APHYNITY (I) both using regularization to control the expressiveness of the neural component. Note that while the original formulation of APHYNITY does not have an inference network to identify or zn (making it intended for data generated from a single parameter setting). we added an inference network (that shares a similar architecture with the other two models) for fairness of comparison.
With focus on identifiability, we focus on the following metrics: 1) for physics- based latent variable Ze, we consider mean squared error (MSE) owing to its physical meaning: 2) for abstract latent variable zn modeled in the neural component, we consider the mean correlation coefficient (MCC) between the true data-generating parameters and zn sampled from its learned posterior density. The calculation of MCC follows standard practice in the identifiability literature [ to evaluate ~ ^ x-identifiablity, we calculate the weak MCC with the method described in 22). where a high MCC provides evidence of successful identification. We also evaluate a more strong identifiable relation ~'p with strong MCC and the details can be found in Appendix E
Metrics:
In addition to metries on Zp and Zx. we further introduce MSE metrics on the generated x in two distinct scenarios: 1) reconstruction MSE measures how well an observed x is reconstructed from the identified Zy and ZN, and 2) prediction MSE measures how well the identified Zp and Zn can be used to generate outside the observed x used to identify them, e.g., for predicting over long time domains, or predicting for different sample that comes from the same generation process but with different initial conditions. We use these two metrics to showcase that good reconstruction does not guarantee good identification, while the ability to use the identified latent variables to predict for different samples (other than observed) may be surrogate for identifiaiblity.
7

---

Lybnn-VVA
Mca-Ilyeni-AE
Iesocntruutiin
1.0
3.35
+03
-1.0
0.
30
52
0.4
033
175
LXB
220
175
013
0.2
150
175
Teers)
Trams
Taato
Forced Damped Pendulum
Double Pendulum
Figure 1: Visual results on synthetic data for reconstruction and prediction performances.
6.1 Results on Synthetic Data
Reconstruction vs. Prediction Performance: Table summarizes quantitative identifiability metrics on the presented meta-hybrid-VAE in comparison its baselines, where the prediction performance of all models is mentioned by their ability once identified to predict for sample sharing the same data- generating parameters but different initial conditions. As shown, meta-hybrid- VAE significantly improved the identifiability of the abstract latent variable zn (the highest MCC close to 1) and via which, moderately improved the accuracy of the physics-based latent variable Zp (the lowest MSE). Very importantly, while both baseline models identified Zp with a reasonable accuracy, their MCC values were significantly lower - providing strong empirical support for the un-identifiabiiiiyy of the neural component of the hybrid-DGM as theorized in this paper.
Also importantly, all hybrid models achieved comparable reconstruction MSE for observed x's. where the meta-hybrid- VAE did not necessarily produce the lowest MSE. The ability of the identified hybrid-DGM to predict for different samples other than observed, however, varied significantly: while the meta-hybrid- -obbdd--iEiid-e- was able to deliver MSE comparable to the reconstruction task, the two baseline models saw a deterioration of MSE by two magnitudes. Fig. Proovies visual examples to demonstrate this performance difference between reconsruction and prediction tasks, stressing the importance to look beyond reconstruction performance for evaluating hybrid-DGMs, and 2) consider prediction of unobserved samples as a potential surrogate for identifiability measures when ground-truth of latent variables not available (real data settings in Section 6.2).
Interestingly. among the three datasets, the advection-diffusion system appeared to be relatively simple identify the baselines did not exhibit significant degradation of performance in the more difficult identifiability metrics (MCC and prediction-MSE) This is potentially because the missing component from the prior physics has small effect. comparison, the MCC and Prediction-MSR of the baseline models were significantly poorer in double pendulum, suggesting difficult missing physics be identified. Regardless, meta-hybrid-Va. -AA was able obtain significant margins of improvements compared to baselines across all datasets.
Predictions over Longer Time Domains: Fig. shows the prediction performance of the three models when predicting the trajectory of pendulum movement beyond the time domain used to identify the model, with visual sample of forced damped pendulum system. As shown, while all models' performance deteriorated as they predicted outside the time domain used in training, the presented meta- meta-hybrid- VAE demonstrated significantly smaller drop in performance and slower rate of deterioration as the prediction horizon increased. This provides strong evidence regarding the importance of the identifiability, even for the neural component, of hybrid model for the purpose of predictive tasks.
OOD Performance: Figure 3 (left) summarizes the performance of the three models in settings where either the physics-based component Zp or the neural component EN was outside the training distribution for forced damped pendulum system. Results demonstrated that, meta-hybrid- VAE obtained the strongest performance in OOD settings for both the identification of the physics and neural components. Details of the OOD parameters can be found in the Appendix A.
Empirical Verification of the Condition for Identifiability: Theorem 1-iv) stipulates that the identifiability of the hyb depends on number of independent conditional densities observed, in relation to the dimension of the latent factors to be identified. To experimentally

---

Ground Truhh
APHYNITY
HyboH-VAE
Meu-RRybod-NAE
APRENTT
Anue
Time Domain
Time Domein
Time Domain
Time
Forced Damped Pendulum
Advection-Diffosse System
Double Pendulum
Visualization Example
Figure 2: (Left) The performance of predictions over longer time domains; (Right) visual example of forced damped pendulum system. The dark part is the training time domain, and the light part is the time domain outside training (longer time domain).
APHYNITT
Hylei-VVA
Meu-HothiiAWW
MSF of
MCC
1D
00D
00Dx,
0OD
x0D
The mmmer of tasks
Figure 3: (left) Results of OOD; (right) Results of the condition for identifiability.
verify this, we evaluated the identifiability metric MCC the hybrid-DGi when learning over data generated from increasing number of distinct parameter values governing the pendulum system (Equation (), i.e., number of tasks in meta-lcarning. The results in Figure 3 (right) showed that the MCC metric exceeded 0.85 when 7 tasks appeared in the training data and stablized when 8 or more tasks appeared. This agreed with the theorem that, for the pendulum system that had 3-dimensional parameter vector be identified in the neural component of the hybrid model with Gaussian assumption of 2-dimensional sufficient statistics, a minimum of 3*2+1 7 distinct "tasks" needed to identify the system.
Additional Alation Analyses of Meta-Hybrid--ar VAE: In Appendix C.I we provided ablation results the pendulum system demonstrate that the identifiability results of the presented meta- hybrid-VAB were minimally affected by the number parameters to be identified, as long as the theoretical condition for identifiability is met. In Appendix C.2, provided further ablation studies to show that the physics-based component did not suffer from the type of un-identifiabilty discussed in this paper the neural component, although the meta-formulation did moderately improve the accuracy of the estimation of the physics-based parameters.
6.2 Real Data of Double Pendulum
Experimental settings: We used the dataset of double pendulum introduced by 44, which contains 21 videos the pendulum. Each run lasts approximately 40 seconds and recorded at 400Hz. We extractd the position of the pendulum limbs from each frame with elementary computer- vision tools. We divided each video into observations of x's to consist of 20 temporal frames with a sampling frequency of 100 Hz. Because there is clear indication of which samples belong to the same data- tataageenratiin process, we use 7 samples preceding the current query sample as the context samples. We generated 10000, 3400, and 3400 training, validation, and test samples, respectively.
We adopted Equation 19 as the physics-component of the hybrid-DGM, using the known lengths of the two arms (L1 91mm and L2 70mm) and assuming m 1. The total energy of the double

---

APRYNITY
BylnidVAE
Mct-Ryannl-VE
-0.20
-0.3
-0.40
03
Teans
TeatS
Reconstruction
Prediction
Figure 4: Examples of reconstruction and generation results real double pendulum data.
pendulum decreases over time in all videos, which is not explained by this physics model. Nor does the physics model consider potential vibrations or errors extracting the arms' positions, which is expected to be learned by the neural component of the hybrid-DGM with the dimension of An 4.
As ground-truth generative factors are not available in the real data, we relied on prediction MSE as a surrogate for the identifiability metric informed by results in Sections [A. While all models delivered comparable reconstruction MSE, meta-hybrid VAE was able to achieve a prediction MSE at a level similar reconstruction while the two baselines experienced a deterioration by 2-3 magnitudes. Visual examples are provided in Fig. This strong performance in real data provides solid evidence for the presented meta-formi foraulaomultiion establish identifiability of hybrid-DGMs.
Results:
7 Conclusions & Discussion
In this paper, we probe the un-identifiability hybrid-DGMs with unconditioned priors, and present meta-learning as novel solution construct identifiabile hybrid-DGMs both results were supported by strong theoretical and empirical evidence. Moving forward, this work can be improved as follows.
Comprehensive ablation studies of the meta-hybrid- VAE are needed to examine the effect of several key hyperparameters, such as the size of k-shot context and the dimension of ZN- The latter was set to the number of true generative factors in synthetic experiments, and its effect model performance needs to be further examined. Note that this setting was identical across baselines, thus it had no effect on the significant margins of improvements seen by meta- meta-hybrid hae--hybbrd---a VAE compared to baselines. Future ablation studies should also delve into the benefit of hybrid-DGMs in comparison to purely physics-based and purely neural DGMs, which we considered out of scope of the current study that focused on establishing the identifiability of hybrid-DGMs both theoretically and empirically.
Regarding the generality of the meta meta-formulation as solution to identifiable DGMs beyond the hybrid formulation, pppenddx(pprrseenteee an initial investigation on the performance of the presented meta-forma cetta-fommulatinn on general non-hybrie VAE on a synthetic dataset of non-stationary Gaussian time- series, comparison to the identifiable- VAE constructed using known class labels to condition the generative model.
Future works could investigate this effect in broader context. Finally, while the formulatmnntnonn foregoes explicitly observed auxiliary variable, it does assume the ability to pair context and query samples from the same data generation process. This is common assumption in meta-learning and, in scenarios where this knowledge is not evident, simply pairing by preceding temporal samples was shown be effective in our real-data experiments. In scenarios where no information is available for pairing, the presented meta-solution will not be applicable.
Acknowledgments and Disclosure of Funding
This study is supported by the National Key Research and Development Program of China(No: the Talent Program of Zhejiang Province (No: 2021R51004), the NIH NHLBI grant RO ROIHL145599 the NIH NINR grant ROINR01830, and NSF grant 0AC-2212548.
10

---

References
[1] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dy- namics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12) 2021.
[2] Naoya Takeishi and Alexandros Kalousis. and interpretable generative modeling. Advances in Neural Information Processing Systems, 11114809-148227
variational autoencoders for robust
2021.
[3] Ziming Liu, Yuanqi Du, Yunyue Chen, and Max Tegmark. mhysics-augmented learning: A new paradigm beyond physics-informed learning. In NeurlPS 2021 Al for Science Workshop, 2021.
[4] Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, and Jörn- Henrik Jacobsen. Robust hybrid learning with expert augmentation. Transactions on Machine Learning Research, 2023.
[5] Zhaozhi Qian, W. R. Zame, L. M. Fleuren, Elbers P, and M. van der Schaar. Integrating expert ODEs into neural ODEs: Pharmacology and disease progression. In NeurIPS, 2021.
[6] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR, 2020.
[7] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[8] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469, PMLR, 2017.
[9] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[10] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, 2019.
[11] Andy Zeng. Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4):1307-1319, 2020.
[12] Florian Golemo, Adrien Ali Taiga, Aaron Courville, and Pierre-Yve Oudeyer. Sim-to-real transfer with neural-augmented robot simulation. In Conference Robot Learning, pages 817-828. PMLR, 2018
[13] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neural- sim: Augmenting differentiable simulators with neural networks. In 2021 / IEEE International Conference on Robotics and Automation (ICRA), pages 9474-9481. IEEE, 2021.
[14] Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning, arXiv preprint arXiv:2001.04385, 2020.
[15] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In ICLR, 2021.
[16] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. In NewrIPS, 2023.

---

[17] Hermanni Hälvä, Sylvain Le Corff. Luc Lehéricy, Jonathan So. Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. In NeurlPS, 2021.
[18] H Halva and A Hyvarinen. Hidden markov nolinear ica: Unsupervised learning from nonsta- tionary time series. UAI, 2020.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv: 14, 2013.
[20] Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyv, Revant Kumar, al. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research, 18(57)11-59, 2017.
[21] Christopher Biship. Pattern recognition and machine learning (information science and statistics). Springer New York, 2007.
[22] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beems Iden- tifiable conditional energy-based deep models based on nonlinear ica, Advances in Neural Information Processing Systems, 33:12768-1277 12778, 2020.
12

---

A Details on Experimental Data Generation
A.1 Forced Damped Pendulum
We generated data from the full physics function in Equation 2 with observed x [f00--.., Piy...; (P. Pc ((iAt), T == 49, At 0.05. We randomly sampled the initial con- dition po E (-1.57, 1.57] (with Po = 0 fixed). Physics parameters [w, E, A, ø) to be identified were varied as follows: E [0.785, 3.14] E E [0.0, 0.8), A E [0, 40] and
[3.14, 6.28]. We generated total 60000 samples and separated them into training, validation, and test sets with 4,0000, 10000, and I, ,,0000 samples, respectively. In all hybrid-DGMs,
considered the first terms in Equation 2 as the partial physics, thus = w and n with a dimension of 3 is expected to recover [s. A. Φ). This setting induces an additive error in the prior physics. In OOD setting, we set OOD Zp as w E
and OOD Nn as & E [0.8, 1.0], while keeping the rest of the parameters unchanged.
A.2 Advection-diffusion System
We generate data from the following full physics function:
aT TT
8T
Ot
Os²
+
Os =
(18)
where is the spatial dimension. The solution T(s, was observed
(To...., Tr], where
T, : [T(0, (,). T8 12)] )² at 1 j is uniformly distributed in space with dimension of 20. We set 7 49, At 0.02, the boundary condition as T(0, T(8maz ()
and the
initial condition as T(s,0) 0) = csin(Ts// (0/mm//mmaa) with E [0.5, 1.5]. Physics parameters to be identified [a, were sampled as: E [0.01, 0.1] and [0.01, 0.1]. We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMs,, we set partial physics as the first two terms of Equation (T8), thus 2p and nn with a dimension of I is expected recover b.
This represents an additive error in the prior physics.
A.3 Double Pendulum
We generated data from the following full physics function:
Gisnn cos( -22) sin)
of cos (os(111-9))+ (223)) (1
d
Li (m-tallllp -v2)
dt
02
(M+1) sin(p: -p:) sin up sin 1n cos(p: -2:) La coe1:: -p2) p42)
La -p))
(19)
where m m1/m2. The observed x := [( 22.0) ..... (4) (2.)).... (PI.T, (2,7) where
(F1,,) (P2,) is the value of a solution at (5 = jål with T = 79, A/ = 0.025, Li 12= 1.0. We randomly sampled the initial condition 901,0: $2,0 E (-1.57, 1.57 (with $1.0 = 22.0 = 0). Physical parameters [G, mi to be identified were sampled as follows: G E [5.0, 15.0) and m [0.55 1.5]. We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMss set partial physics to be Equation (19) with m = 1, thus &p = G and eN with dimension of I expected to recover m. This induces a multiplicative error in the prior physics.
B The proof of Theorem 1
Theorem 1: Assume we observe data sampled from Px up, pp.xx(D)) defined in Equations (5) with parameters @ = (0, TTAc). Assume the following holds:
(i) The set {x E (x) = 0} has measure zero, where Pe is the characteristic function of the density Pe defined Pe(x - Folfr, fNy: zp, zw])-
(ii The hybrid mixing function Fe is injective.
(ill) The sufficient function Tig are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero.
(iv) There exist de + 1 distinct context sets Dú, D Dār such that the de x de matrix L defined as follows is invertible:
L (A(D-il)) - X(Dr,de) x(DDa.1)
(20)
13

---

+ Then the parameters Φ - (0, (.A)) are -A-identifiabl..
Proof: This proof mainly refers to [6. For simplicity, we denote Fo as f, Fo as f, d₂ x and dg as A. Suppose we have two of parameters (f, T) and (f, T, such that Pr.T.A(x(D"))
(x(()) for all pairs (x, D*). Then:
Pr (x 2x
(21)
/ Pr. (r.A((D")p<(xx f(z))dlz
/ P i DD' D')p=(x f(z))dz
(22)
PT PT-2(- (x) (x)po(x 8)d8 / PTA (x) (x)
x)dx
(23)
| (X)pE- -))dx
: x)dx
(24)
JRd
(PTAA * px) (Pi.X...D (x)
(25)
((PAA :' (w)
(26)
FIPTAFED = Fl
(27)
( = P...AF.D= (x)
(28)
By taking the logarithm on both sides equation 28 and replacing PT, by its expression from
we get:
log x +
(log (x)(+((x)) log ((()))
(29)
log volJz 1x) +
(log (J. - log (D())) + (x)),
Let Dú, Di...., Dāe be the context sets provided by assumption (iv) of the Theorem, and define A(Dī) = X(DF) A(D6). We plug each of those Dj in (29) to obtain de + 1 such equations. We subtract the first equation for Dj from the remaining de equations to get for | .e.
(x)) + +Σ log (D(D))
(x), + +Σ log (,(
(x(D))
(30)
Z,(D))
Let L be the matrix defined in assumption (iv), and L similarly defined for x. Define by 2.(P)).,(P)) >; log Z.(P))Z:(P)) and b the vector of all by for I = L...., de. Experssing equation (30 for all points u, in matrix form, we get:
(31)
We multiply both sides by the transpose of the inverse of L from the left find:
(x)= AT(F-'(xx) (x)) +c
(32)
where = L-T and --
Now by the definition T and according to assumption (iii), its Jacobian exists and an de x matrix of rank d. This implies that the Jacobian of T o f- exists and is of rank d and so is A. We distinguish two cases:
(1) If 1, then this means that A is invertible (becuase A is x d).
f (x) and Txx))) (T,,, (x,), (x.),.... ... Tie (x:)). According Lemma 3 in [6] for each i [1, [1,... dl there exist points x! ..... x such that (T) ... (x()))) are linearly independent. Collect those points into vectors (xl ..... x ), and concatenate the Jacobians J
(2) lf 1, define
14

---

Table 2: The results of the different dimensions of unknown parameters
The number of
MSE zp
MCC 1
MSE of x (Rec)
MSE of x (Pre)
parameters
2
1.22(0.08)ee22
0.97(0.01)
1.22(0.08)e-2
2.02(0.13)e-2
0.99(0.00) 3.62(0.01)e-2
1.29(0.05)e-2
5.01(0.000e=2
4
1.59(0.07)e-2
0.99(0.00)
2.9T(0.06)e-2
6.85(1.11))-2
Table Comparison results of Non-meta and Meta
Method
MSE of Zp
1CC
MSE of (Rec)
MSH of (Pre)
Mon-Meta
9.13(3333)-3
0.99(0.00)
3.28(0. 17)e-2
8.82(2.03)e-2 3.37(0.00)e-2
Meta Zp
3.63
0.99(0.00)
3.18(0. 19)e-2
evaluated each of those vectors horizontally into the matrix Q (JT(x"), JT (xx)) (and similarly define as the concatenation of the Jacobians of (fx)) evaluated those points). Then the matrix Q is invertible (through combination of Lemma 3 and the fact that each component of T is univariate). By differentiating equation (32) for each x', we get (in matrix form): Q = AQ.
The invertibility of Q implies the invertibility of A and Q.
Hence, equation
32 and the invertibility of A mean that (f, ~A (f,
C Alation Analyses of Meta-Hybrid--AE
C.1
Effeet of the Dimension of Unknown Parameters
Here, we provided ablation results on the pendulum system to demonstrate that the identifiability results of the presented meta-hybrid-VAE were minimally affected by the number of parameters be identified, as long as the theoretical condition for identifiability is met. The results can be seen in Table 2
C.2 Meta-component cp
Here, we provided further ablation studies to show that the physics- based component did not suffer from the type of un-identifiabilty discussed in this paper the neural component, although the meta- formulation did moderately improve the accuracy of the estimation the physics-based parameters. The results be seen in Table
D Results on General Non-Hybrid VAEs
We compare our and identifiable VAE (iVAE) synthetic datasets about non-stationary Gaussian time-series in 6 To evaluate the performance of the method, we compute the MCC between the original sources and the corresponding latent samples from the learned posterior. The results can be seen Table 4
of the non-hybrid VAE (Meta- compared to the original VAE
Table 4: The results on General Non-Hybrid VA
VAE
iVAE
Meta-Val VAE
MCC
0.67(0.04)
0.91(0.03)
0.88(0.00)
E
Strong MCC results
We also evaluate proposed model on a more strong identifiable relation ~pp defined as follow:
Definition 3: Let ~ be the equivalence relation & defined the same 16
(0, (0, ~ ( + C T(F (x)) AT(F (x)) +x. X
(33)
15

---

Lp
Z
.11
N2
33
33
42
4
Hybrid-VAE
Meta-HybriddVAE
Figure 5: Visual samples of learned latent variables between different runs.
Table The results of strong MCC
Forced Dumped Pendulum
Advection Dooecction-iifussiiie Diffusion System
Double Pendulum
Strong MCC
Strong MCC
Strong MCC
Strong MCC
Strong MCC
Strong MCC
Ground Truth 0.38(0.02)
Diffcrent Runs (.81(0.08)
Ground Truth
Different Runs
Ground Truth
Different Runs
APHYNITY
0.86(0.05)
0.830.04)
0.5000.00)
0.73(0.00)
Hybrid- VAE
0.29(0.00)
0.80(0.01)
0.80(0.12) 12)
0.79(0.11)
0.50(0.00)
0.72(0.00)
Meta Hybnd-VAr VAE
0.55(0.02)
0.90(0.06)
0.97(0.03)
0.96(0.00)
0.96(0.00)
LOKK0.00)
If is block permutation matrix, we denote this relation by rpp-
The theory according to ~pp is following:
Theorem 2: (e 2 2) Assume the hypotheses of Theorem hold and that e 2. Further assume:
(2.i) The sufficient statistics Ti. in Equation 15 Jare twice differentiable.
(2.ii) The mixing function Fo has all second order cross derivatives. then the parameters (6, T,)c) are r-p-identifabbe.
Theorem 3: (e Assume the hypotheses of Theorem I hold and that e = 1. Further assume:
(3.i) The sufficient statistics Tr, not monotonic.
(3.ii) All partial derivatives of Fo continuous.
then the parameters (0, T, Ar) are ---identifiabl.
The proof of Theorem and can found in 66. Then to evaluate the performance on ~>p, we calculate the strong MCC between leamned latent variables and ground truth variables (Strong MCC Ground Truth) and strong MCC between runs of the model (Strong MCC Different Runs) 6 22
The results be found in Table 5 and provide visualization example of forced damped pendulum system to show the relation learned latent variables between two different runs.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our method and experiments have show our main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed it on Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
The proofs of the theorem in the main paper can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided enough information to reproduce the results in Section and Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[NA]
Justification:
The data and code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cessiinnooccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp..cppulli//uuuii wwwtc//ggui////eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental settings and details on Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the standard deviation of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::///euriipee http://......p....ttttcccccieeea
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:[NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "yBGgP3TL2q", "reviewer": "Reviewer_1aic", "summary": "The original contribution for his manuscript is a metalearning framework to identify the physical parameters in a hybrid deep generative models.", "questions": "The proof-of-concept results on pendulum models and the reaction-diffusion equation have very few parameters to identify and do not inform that the method would be useful in more realistic or more complex models. Could you provide examples with more parameters to identify? Could you give theoretical support to how fast your method would converged to the identified quantities of interest as the number of parameters to identify increases?", "limitations": "Limitations are discussed in the text.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "This article starts with a good question about the identifiability of physical parameters in hybrid deep generative models (that include physics as an inductive bias). Solving this question could answer key questions in data-driven knowledge discovery. This article presents  meta learning framework which could help identify the physical parameters statistically.", "weaknesses": "This article seems to be a work in progress. The text alone is not an end product. Symptomatically of this work in progress, the central concept of this article of identifiability is misspelled more than 10 times throughout. The contributions of this paper are not clearly stated and hard to find. It seems that the theory stated in this article was already published elsewhere, as illustrated by the reference to another paper for the proof of the only theorem of this article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1aic for the critical comments. Please see the overall response 4 (and the corresponding results in the attached pdf) for clarification regarding the effect of the number of parameters to be identified, and the overall response 1-3 (and the corresponding results in the attached pdf) for the clarifications on the contribution of the presented work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzfs4ybHjm", "reviewer": "Reviewer_1hrQ", "summary": "A method and a theory of learning hybrid deep generative models (esp., VAEs) are proposed. The method is based on meta-learning, and the theory is about the identifiability of the neural component's latent variable. The method is also empirically compared to baseline methods of hybrid DGMs.", "questions": "Please elaborate on the point (1) in the Weaknesses section above.", "limitations": "Limitations nicely discussed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- In the introduction, the motivation for analyzing the identifiability of hybrid DGMs is clearly stated.\n- The potential connection to the study around nonlinear ICAs sounds reasonable and interesting.\n- The method is technically reasonable. To my knowledge, meta-learning has not clearly been applied in the context of hybrid modeling.\n- Experiments are done with reasonable, yet mostly synthetic, datasets and with adequate baseline methods.", "weaknesses": "(1)\nit is unclear what is the theoretical contribution particularly by this work. The theory in Section 5 looks like a mere rehash of what was presented in [6]. Also, the theory in Section 5 focuses only on the neural network part of hybrid DGMs, and the presence of the physics part of the model does not seem to affect the discussion. This limits the significance of the theoretical contribution.\n\nMoreover, as the authors pointed out, the un-identifiability of hybrid DGMs stems both from 1) the un-identifiability of the neural component and 2) the over-powering effect of the neural component. The authors say that they focus on 1), but then the discussion becomes simply about non-hybrid, general DGMs, which makes the authors' claim that they theoretically analyzed the identifiability of *hybrid* DGMs questionable.\n\nMinor points:\n- In Eq. (13), the prior of $z_P$ is missing.\n- What makes the synthetic and real double pendulum data different? E.g., is there supposed to be any source of non-negligible noises?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1hrQ for the constructive comments. \n\n1. Contribution:  Please see the overall response 1 for our rationale for not focusing on the identifiability of the physics-component, response 2 for the significance for the (un)identifiability of the neural component that we focused on, and response 3 for the theoretical contribution of our meta-formulation of identifiable-DGMs in general (applies to hybrid or non-hybrid DGMs). We have added new results in the attached pdf to support each of these responses. \n2. Difference between synthetic and real double pendulum data: In the synthetic pendulum data, we know that the source of error in the prior physics is due to missing the friction and external force terms. In the real double pendulum data, the true governing physics  is unknown. The sources of unknown factors include the unknown masses of the two pendulums (assuming to be equal), presence of friction and its associated parameters, potential vibrations or errors in extracting the arms’ positions from the videos, as described in [5] as cited in the manuscript. \n3. Typo in Equation (13): Thanks for pointing this out. The prior term of $p(z_p)$ is missing. We will add it in the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Bmwa9J9pi", "reviewer": "Reviewer_HAHc", "summary": "This paper discusses the identifiability of Hybrid Deep Generative Models and proposes a Meta-Learning-based approach to address the (un)identifiability issue of current DGMs. Theoretical discussions are provided, and experiments are conducted to verify the proposed method.", "questions": "1. In Definition 2, what is $T(\\cdot)$? Is that the sufficient statistic? Please define it explicitly.\n\n2. In Theorem 1, in assumption (ii), what is $\\mathcal{F}_\\theta$? What is the difference between $\\mathcal{F}_\\theta$ and $\\mathcal{F}(f_P, f_{N_\\theta}; Z_P, Z_N)$?", "limitations": "The major limitation is the motivation of this work, as the reviewer mentioned in the weakness section. Another weakness is the lack of technical contribution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper adopts the result from [1] gives a discussion on the indetifiability issue of Hybrid Deep Generative Models's data driven part.\n\n\n[1] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.", "weaknesses": "1. In \"Augmenting Physical Models with Deep Networks,\" identifying the physical part and the data-driven part is important, as mentioned in previous work such as [1]. As mentioned by the authors, this paper focuses on the identifiability issue of the latent variables and the model parameters of the data-driven model. However, the motivation for identifying this part is unclear since it does not have any physical interpretation. The main goal of the data-driven part is more about approximation accuracy rather than interpretability (identifiability). Therefore, the reviewer is not fully convinced by the motivation.\n\n2. The authors also agree that there are two sides to the identifiability issue: (1) physical parameters and parameters from the data-driven part, and (2) the identifiability of the parameters in the data-driven part. This paper focuses on the latter. However, assuming there is no identifiability issue with (1), the problem degrades to the same setting as [2]. Therefore, the theoretical discussion seems borrowed from paper [2], without any revisions.\n\n3. The writing needs to be further improved, including the definition of notations.\n\n[1] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124012, 2021.\n\n[2] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer HAHc for the constructive comments. \n\n1. Contribution:  Please see the overall response 2 for the significance for the (un)identifiability of the neural component that we focused on, and response 3 for the theoretical contribution of our meta-formulation of identifiable-DGMs in general (applies to hybrid or non-hybrid DGMs). We have added new evidence in the attached pdf to support each of these responses. \n2. Yes, T denotes the sufficient statistics.\n3. Both $\\mathcal{F_\\theta}$  and  $\\mathcal{F}(f_P, f_{N_\\theta}; z_P, z_N)$ denote the hybrid generative/mixing function.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXRhBBUTvQ", "reviewer": "Reviewer_5WN4", "summary": "The paper thoroughly assesses the issue of identifiability of both the neural and physics components in hybrid deep generative models (hybrid-DGMs), and proposes a novel approach to formulate such models using meta-learning. The authors show the performance of the model in comparison to other hybrid-DGMs and the ground truth for three well-known physics examples.", "questions": "How resource heavy this method is compared to the other methods used in the comparison?\nHow well does it perform for OOD testing in terms of robustness and generalisation?\nHow well it does predict the behaviour of dynamical systems over longer time intervals (is there a performance degradation over time)? \nWould meta learning using the Wasserstein distance instead of the KL divergence improve the identification of both the physics and the neural components?\nAs suggested by the authors themselves this work would benefit from in-depth hyperparameters to performance study, particularly taking into account the relationship between data samples and model performance.\n\nComments:\n- as a minor point - there are some references that didn't compile as well as typos in the manuscript.\n- another minor point is that the appendix is missing with the default NeurIPS instructions left in place - this can be removed.", "limitations": "The authors didn't provide a separate limitations section but discussed the limitations of their work and future directions in the Discussion/Conclusion section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is the first theoretical study of the identifiability of hybrid-DGMs.", "weaknesses": "Possible limitations of applicability related to specifics of meta-learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5WN4 for the supportive and constructive comments. \n\n1. Resources:   On the same device and dataset, Hybrid-VAE takes 250.24s for 50 epochs and Meta-Hybrid-VAE takes 291.68s. In all experiments, Meta-Hybrid-VAE requires approximately 0.5 times more epochs to converge.\n\n2. Performance over longer time intervals: We thank the reviewer for this suggestion. Please see the overall response 3 and the added results in the attached pdf. The results showed that the presented identifiable hybrid-DGMs are significantly more robust in its performance of predicting over longer time intervals (while the non-identifiable baselines deteriorated rapidly over longer time intervals). \n3. Performance in OoD data: We thank the reviewer for this suggestion. Please see the overall response 3 and the added results in the attached pdf. The results showed that the presented identifiable hybrid-DGMs are significantly more accurate than the non-identifiable baselines in data samples that are out of distribution either due to the physics-component or neural component. \n4. Additional ablation studies: In the manuscript we showed that the meta-formulation is critical to the identification of the neural component. In the added results in the attached pdf (Table 1), we further ablated and showed that the meta-formulation can improve but is not critical to the identification of the physics component. In the added results in Table 3, we empirically ablated the effect of the number of distinct “tasks” (i.e., distinct parameter distributions) on the presented method, through which we empirically verified the theoretical condition for identifiability established in Theorem 1-iv). In the added results in Fig 4, we further added ablation studies to show the effect of the number of parameters to be identified on the presented method. \n5. Wasserstein vs. KL distance: This is an excellent question which we will investigate in our future studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We clarify the reviewers’ main questions about the theoretical contribution of this work as follow:
1. **Identifiability of the physics-based component in hybrid-DGMs**: The reviewers questioned our motivation for focusing on the un-identifiability of the neural (instead of the physics) component. We do not focus on the physics component because 1) existing works (e.g., [1-2,5] cited in the manuscript) have proposed promising solutions in accurate estimation of the physics parameters , and 2) the physics-component does not suffer from the type of theoretical un-identifiability as the neural-component does – as shown in **Table 1** of the attached pdf, with or without the meta-formulation, the estimation of physics parameters saw good accuracy. In comparison, the identifiability of the neural component has a much more significant effect on the performance of the hybrid-DGM (Table 1/Figs 1-2 in the manuscript). Regardless, as a side benefit of improved identification of the neural component, our approach did result in a higher accuracy of the physics parameter estimation as well (Table 1). 
2. **Un-identifiability of the neural component in hybrid-DGMs**: We respectfully disagree with Reviewer HAHc on the (in)significance of the identifiability of the neural component. The goal of a hybrid-DGM is to use the data-driven part to learn the “missing physics” (those not explained by our prior knowledge) from data – to correctly identifying this missing component, therefore, is critical even if the model itself is a neural-net (for the same reason, there is an active body of research on the identifiability for non-hybrid DGMs even though the model is data-driven). Left unaddressed, many different solutions of a hybrid-DGM can fit the reconstruction accuracy equally well, but many – due to the non-identifiable neural component – **will have minimum to no predictive power** as a hybrid-DGM: in addition to our strong results in Table 1/Figs 1-2 in the manuscript, we add additional evidence (thanks to Reviewer 5WN4’s comments) to show that the un-identifiable neural component will significantly reduce the ability of the hybrid model to predict over longer time intervals (**Fig 1** in the attached pdf) or to perform in OoD situations (**Fig 2** in the attached pdf). We believe these results (with theoretical support) provide strong evidence about the significance of constructing identifiable neural components in a hybrid-DGM – a critical gap in the current literature. 
3. **Meta-learning as a formulation to construct identifiable DGMs**: We acknowledge that the presented theory heavily draws on [6]. As a seminal work in nonlinear ICA and identifiable DGMs, however, [6] serves as the theoretical foundation for a line of follow-up identifiability theorems [a-c] where, similar to the presented work, the focus is on how to construct conditionally independent generative models based on the theory in [6]: in most existing works, this conditioning leverages **observed** auxiliary variables such as class labels. In this regard, our work is **the first** to theoretically show that the meta-formulation of DGMs enables the construction of conditionally independent models, with theoretically-proven conditions for their identifiability (see Theorem 1). This connection, though may now appear intuitive given the Theorem presented in this manuscript, is a novel contribution to general non-hybrid DGMs that has never been constructed in the existing literature. In **Table 2** of the attached pdf, we add identifiability results of the meta-formulation of a non-hybrid VAE compared to the identifiable-VAE in [6] that uses class label as the auxiliary variable.
[a] Klindt et al, Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding, ICLR 2021
[b] Halva et al, Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA, NeurIPS 2021
[c] Yao et al, Temporally Disentangled Representation Learning, NeurIPS 2022
4. **The number of parameters to be identified**: Reviewer 1aic questioned about the identifiability of the presented hybrid-DGMs as the number of parameters increases. We would like to clarify that 1) The number of parameters considered in our work followed the standard practice used in published hybrid-DGMs (see [1-2, 5] cited in the manuscript: the maximum number of unknown parameters to be identified was 3 in [1],4 in [2], and 3 in [5]), and 2) Theorem 1-iv) in the manuscript did already specify the identifiability condition for the presented model dependent on the number of parameters to be identified: for a d-dimensional parameter vector following exponential families of distributions with e-dimensional sufficient statistic, its theoretical identifiability is guaranteed as long as we have observations generated from a minimum of de+1 unique parameter distributions (regardless how large the value of d may be) –  in **Fig 3** of the attached pdf, we empirically verify this theoretical condition on the Pendulum system with 3 unknown parameters in the neural component.  Additionally. In **Table 3**, we show that the performance of the presented hybrid-DGM is minimally affected by the increase in the number of parameters to be identified (on the Pendulum system). In **Fig 4**, we add results from a complex system of positron emission tomography (PET), where activity images x are generated from radiotracer kinetics governed by 2-tissue compartment model with pixel-wise kinetic parameters: the data-generating compartment model has 4 unknown parameters in each region of interest (ROI), with 5 ROIs in total. The meta-hybrid-DGM (using simplified 1-tissue compartment model as the prior physics) demonstrated strong identifiability results as measured by the MCC.

Meta-review: The manuscript discusses the identifiability of hybrid deep generative models, and provides theoretical and experimental evidence that hybrid DGMs with unconditional priors are unidentifiable, but that identifiable hybrid DGMs can be constructed via meta-learning.

All reviewers found the topic interesting, but found the focus on the identifiability of the data-driven component and the experimental focus on simple physical systems with few numbers of parameters limiting. Since the authors responded to these points adequately during the rebuttal, the reviewers accordingly raised their scores and eventually recommend acceptance. Since some of the reviewers also criticized the presentation of the paper, I strongly suggest to revise the text and structure for the camera ready submission.
Decision: Accept (poster)