Med-Real2Sim: Non-Invasive Medical Digital Twins using Physics-Informed Self-Supervised Learning

Authors: keying  kuang
frances  dean
jack b jedlicki
david  ouyang
anthony  philippakis
david  sontag
mit  ahmed
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:16:28

Abstract: A digital twin is a virtual replica of a real-world physical phenomena that uses mathematical modeling to characterize and simulate its defining features. By constructing digital twins for disease processes, we can perform in-silico simulations that mimic patients' health conditions and counterfactual outcomes under hypothetical interventions in a virtual setting. This eliminates the need for invasive procedures or uncertain treatment decisions. In this paper, we propose a method to identify digital twin model parameters using only noninvasive patient health data. We approach the digital twin modeling as a composite inverse problem, and observe that its structure resembles pretraining and finetuning in self-supervised learning (SSL). Leveraging this, we introduce a physics-informed SSL algorithm that initially pretrains a neural network on the pretext task of learning a differentiable simulator of a physiological process. Subsequently, the model is trained to reconstruct physiological measurements from noninvasive modalities while being constrained by the physical equations learned in pretraining. We apply our method to identify digital twins of cardiac hemodynamics using noninvasive echocardiogram videos, and demonstrate its utility in unsupervised disease detection and in-silico clinical trials.

Main paper: 
 introduction: 
 With increasing health data availability, there is excitement about refining physics-based models of human body systems to be patient specific, as personalized physical models can provide the basis for in-silico experiments and timely diagnosis (27; 31; 55; 25) . This personalized vision of healthcare has given rise to virtual patients constructed with data-tuned models, known as digital twins. Originally an engineering concept, digital twins have recently been realized as a resource in healthcare (6; 4) . The concept combines data-driven approaches with mechanistic or simulation techniques, serving as a bidirectional map between real data and simulations. Medical digital twins have been applied and developed broadly in applications ranging from cellular mechanics (24) to the development of whole body and human digital twins (50; 76) , with aims ranging from tailoring cardiovascular interventions (30) to agent-based trauma care management systems (14) . Both academic and industry efforts are prolific with digital twins having been developed for various medical problems (9; 69; 74; 13; 22; 70) .In this paper, we propose a framework for identifying patient-level digital twins using noninvasive medical images. We focus on scenarios where identifying a patient's digital twin from noninvasive data allows us to simulate other physiological parameters that typically require invasive measurement, such as through catheterization. Such an approach assumes that there is a mapping from rich noninvasive imaging data to patient-specific mechanistic models of physiology; such a mapping is not known to scientists humans but can be learned from data. This hypothesis is supported by previous experimental work (2; 64; 43) . However, the challenge lies in the absence of datasets containing both noninvasive data and the corresponding parameters of the underlying physics-based physiological model, hence the problem cannot be solved using standard supervised learning. To this end, we propose a new class of physics-informed neural networks (PINNs) (60) to incorporate inductive biases informed by mechanistic models of patient physiology into neural network architectures.Summary of contributions. We study obtaining latent parameters of a physics-based model from image measurements directly as an inverse problem. Learning patient specific parameters for a physics-based model from a complete description of patient physical states and a known forward model is an inverse problem. This task can be modeled using, for example, a PINN approach (71; 39; 29) . The process of obtaining patient physical states from non-invasive medical imaging is a second inverse problem with an unknown forward model. Obtaining patient specific latent parameters from non-invasive medical imaging directly combines these two inverse problems and could be modeled by learning the unknown process components. Large numbers of labeled data pairs of images and parameters for the full inverse task would need to be obtained from invasive procedures to use supervised learning. Instead, we propose a training approach that structurally resembles selfsupervised learning (SSL), and leverages labeled data pairs of images and non-invasively measurable patient physical states. We first train a neural network using supervised learning with synthetic data to approximate the forward physics-based model as a pre-text task. We then freeze this network to finetune the learning of a solution to the composite inverse problem from the labeled measurable data pairs. Our methods are then extendable to include exogenous intervention parameters for the development of in-silico experiments. We call this framework "Med-Real2Sim" since it learns virtual simulators of patient phyisology on the basis of real (noninvasive) data.Noninvasive dataWe apply our methodology in the setting of predicting patient specific cardiac pressure-volume loops from echocardiography. The relationship between left-ventricle pressure and volume is an important description of cardiac function (36) . Pressure-volume catheter measurements are highly invasive, challenging their routine clinical use (8; 1; 77) . Cardiac ejection fraction, the ratio between left-ventricular stroke volume and end-diastolic volume, while more routinely measured as a proxy for cardiac function, does not take into account cardiac pressure, preventing diagnosis of many severe cardiac conditions and providing motivation to develop digital twins for pressure-volume relations (80; 45; 59) . Cardiovascular pressure and volume are commonly modeled using hydraulic analogy models, representing the hydraulic system by electronic circuits, characterized by lumped parameters corresponding to system attributes. Systems of differential equations describe the relationships between circuit parameters and volume and pressure states. A model developed in (66) relates left-ventricular pressure and volume with and without the addition of a left-ventricular assistance device (LVAD). We train a neural network to learn to map between parameters and solution states of this model and use this learning to fine-tune a convolutional neural network that learns patient specific model parameters using echocardiography frames.2 Method 
problem setup: 
 We consider a physics-based model M : Θ → X of a physiological process which maps a set of parameters θ ∈ Θ to a set of states x ∈ X . The parameters θ describe the physiological process in an individual patient; each patient i is associated with a patient-specific parameter set θ i and physical states x i . We are interested in a setup where the complete set of parameters and states can only be measured invasively, e.g., through a catheterization procedure. We conceptualize the physics-based model M(θ i ) as the "Digital Twin" of patient i. That is, if we have oracle access to the parameters θ i of patient i, then we can simulate their physiological states through the following forward process:EQUATIONThe model M is assumed to be known and would typically be defined by a system of differential equations written f θ (x) = 0 with a physically interpretable parameter set θ. One can use numerical methods to solve f θ (x) = 0 for x and construct the solution M(θ i ) for each patient parameter set θ i but a closed form solution to define the map M may not exist even though the model is known, such as in the case where f θ = 0 has no closed form solution. The level of granularity in simulating the (true) physiological process depends on the fidelity of the physics-based model M.In addition to modeling the natural state of the physiological process, we also consider physics-based models of clinical interventions (e.g., medical devices). To model an interventional process, we consider an augmented version of the physics-based model M : Θ × U → X , which maps the endogenous parameters θ ∈ Θ of the patient and an exogenous intervention u ∈ U to a physiological state x ∈ X . The augmented model describes how the intervention u alters the physiological states of a patient with parameters θ-in the causal inference literature, this class of models is known as structural causal models (SCMs) (57) .To identify the model M(θ) traditionally and simulate the states x, one would need to conduct a number of invasive procedures for each patient. Here, we assume that we have access to a measurement y i for each patient i, which reflects some aspects of the underlying physiological process. We assume that y is acquired through a non-invasive procedure that might be conducted routinely in clinical practice (e.g., an electrocardiogram). The relationship between the physiological state x and the non-invasive measurement y is given byEQUATIONThus, by combining ( 1) and ( 2), we arrive at the following relation between the parameters of the physics-based model and the non-invasive measurements:EQUATIONThat is, the relation between the non-invasive measurements and the physical model parameters is a composition of two functions; an unknown map K that describes the relation between the physiological states and the observed measurements (e.g., the relation between pixels in an imaging modality and underlying health states), and a known mathematical model M that captures the relation between the physiological states and unknown parameters θ. Our goal is to develop a learning algorithm to identify the digital twin M(θ i ) for patient i given their non-invasive measurement y i .Digital Twin Modeling as an Inverse Problem. Note that obtaining patient-specific parameters θ i from physical states x i amounts to solving the following inverse problem:EQUATIONHowever, we do not directly observe the state x, but rather a measurement y obtained by passing x through the unknown forward model K. Thus, obtaining the physical parameters from y entails solving the composite inverse problem given by:EQUATIONSolving this problem entails two challenges. First, we do not know the forward model K. Second, we cannot directly learn the inverse map F -1 using supervised learning since this requires access to a labeled dataset {(y i , θ i )} i . Obtaining such a dataset would require conducting a large number of invasive procedures, which we seek to avoid.An alternative way to learn the inverse map F -1 is to break down the composite inverse problem in (5) into separate inverse problems, and only learn the inverse of the unknown map K -1 using examples of the form {(y i , x i )} i by fixing the map M. If we can construct a mapping representative of M from any patient parameter set θ ∈ Θ to state x ∈ X , then as we can write K -1 as follows:EQUATIONwe can learn F -1 using training samples which only train K -1 .As we will ultimately use supervised methods and gradient descent to learn Eq. ( 6), we need a method for fixing M that allows for mapping any arbitrary parameter set θ to corresponding state x. There are several methods in the literature for solving the forward dynamics of M that we might think of to construct this mapping. These include physics-informed neural networks (PINNs) (60) , which train a network to approximate the solution x i to f θi (x i ) = 0 thereby solving M(θ i ) for a fixed patient parameter set θ i . Traditional PINNs learn the function x i by incorporating the differential equation f θi = 0 as a soft-constraint in the network's training loss alongside a data loss. In the setting of a ordinary differential equations, Neural ODEs (10) train a network to learn the differential equation operator f θi on the states x i to solve for x i = M(θ i ) using differentiable ODE solvers. These methods can learn M(θ i ) for a fixed patient parameter set θ i and can be compared to the use of a numerical differential equation solvers.These existent approaches for solving M, however, only give the states x for a single patient at a time. To learn K -1 by training a network through the map M, we need a method that approximates the map as otherwise, we would need to train infinitely many separate PINNs or Neural ODEs to cover the entirety of a plausible parameter range potentially traversed in supervised learning of (6) .Non-Invasive Data Acquisition. While we do not have direct access to the patient physiological state x i to train a model to learn K -1 , the noninvasive observation y i typically measures physiological quantities that can also be derived from the true state x i . We assume that x is a physiological variable that can be derived from both x and y, i.e.,EQUATIONwhere m(.) is a known function and g(.) is a labeling procedure, typically conducted manually by a physician or automatically through a built-in algorithm in the data acquisition device. This physiological variable represents a quantity of interest assessed by the noninvasive modality (e.g., breast density in mammography, blood pumping efficiency in ultrasound). These variables establish a connection between the true state x and the observation y, as they can be both simulated from the physical model and measured based on y.Given the setup above, our digital twin modeling problem can be formulated as follows. Given a dataset of noninvasive measurements for n patients, D = {y i } n i=1 , and a physiological variable of interest xi = m(x i ) = g(y i ), our goal is to train a model that can identify the underlying physics-based twin M(θ n+1 ) for a new patient n + 1 based solely on their corresponding noninvasive measurement y n+1 . We seek to do this by fixing M and learning K -1 from the data D = {(y i , xi )} n i=1 .Given the training dataset D = {(y i , xi )} n i=1 , a typical supervised learning task is to train a model to predict the physiological variable xi on the basis of the noninvasive observation y i via standard empirical risk minimization (ERM):EQUATIONThe motivation for training the model f is usually to automate the collection of physiological variables from medical images in a clinical workflow (68) . These models follow a fully data-driven approach and do not incorporate our knowledge of how the physiological processes being measured function on the biological or physical levels. One could think of the supervised model f as a data-driven model of the function g(.) in (7) .Recall that our goal is to recover the latent physical parameters θ that underlie the physiological processes that generated the observation y. If we had oracle access to a dataset of the formD * = {(y i , θ i )} n i=1, then a supervised solution to identifying the digital twin would be:EQUATIONThe solution to (9) provides an approximate solution to the inverse problem in (5) . As we lack access to D * , we can only learn the solution to (5) using the observed dataset D. To this end, we propose a two-step approach for learning the parameters θ from y, leveraging the structural similarity between the composite inverse problem in (5) and the pretraining/finetuning paradigm in self-supervised learning (SSL). An illustration of the two steps is provided in Figure 2(a) .Step 1: Physics-Informed Pretext Task. In this step, we pretrain a neural network to imitate the forward dynamics of the physics-based model, i.e., x = M(θ). We do so by first sampling ñ synthetic training examples from the physics-based forward dynamics as follows:EQUATIONWe denote this synthetic dataset by D = {( θj , xj )} j . Next, we use this dataset to train a feed forward neural network on the pretext task of predicting the patient physiological states from the physical parameters using ERM as follows:EQUATIONThis training process distills the true physics-based model, M(θ), into the weights of a neural network ϕ M . That is, the forward pass of the network will emulate the forward dynamics of the physiological process, i.e., x ≈ ϕ M (θ).Step 2: Physics-Guided Finetuning. Given the observed dataset D = {(y i , xi )} i , we train another model to predict the physical parameters θ from the observed measurements y using the loss functionEQUATIONHere, the model pretrained on synthetic data from the physical simulator, ϕ M , is frozen and only the model ϕ F is finetuned using real data on y and x. The neural network trained in Step 2, ϕ F , represents an approximate solution to the composite inverse problem in (3), i.e., ϕ F (.) ≈ F -1 (.).For a new patient n + 1, we discard the pretrained model ϕ M , and use the model ϕ F to predict the patient's digital twin based on their noninvasive measurement y n+1 as follows:EQUATION)Interpretation of Physics-Informed SSL. Our proposed algorithm structurally resembles the SSL paradigm, where, akin to SSL, we decompose our model into a "backbone" architecture and a taskspecific "head" (47) . However, there are fundamental conceptual differences between standard SSL approaches and ours. In standard SSL, the backbone is a high-capacity model pretrained on a pretext task using unlabeled data to derive a general-purpose representation transferable to many downstream tasks, while the low-capacity head is finetuned for the specific task. In our physics-informed approach, SSL serves to constrain rather than enhance flexibility. We first pretrain the low-capacity head to distill the laws of physics, and then finetune the high-capacity backbone model to learn a mapping that aligns with the frozen head to produce the observed physiological variables x while respecting the physical laws. In doing so, we force the backbone model to learn a mapping from the measurement y to the physical parameters θ.The concept of distilling a physics-based dynamic model into a data-driven one draws from engineering methodologies that develop surrogate models to simplify computational complexity using simulated data (e.g., (58) ). This data-driven approach mirrors other supervised learning generative strategies for solving inverse problems or system identification with known dynamics in control theory (51; 78) . Importantly, physics-informed SSL scales efficiently with increasingly complex and higher fidelity physical models, as its computational requirements involve one offline process (Eq. ( 10) and ( 11)) once the synthetic data is acquired. 
digital twins for cardiovascular hemodynamics: 
 We apply our physics-informed SSL method to perform patient-specific Pressure-Volume (PV) loop analysis in cardiology using echocardiography (cardiac ultrasound). Left ventricular (LV) PV loops illustrate the LV pressure against LV volume at multiple time points during a single cardiac cycle, providing a comprehensive view of a patient's cardiac function and encoding various hemodynamic parameters like stroke volume, cardiac output, ejection fraction, myocardial contractility, etc., which can be used to diagnose cardiovascular diseases (7) . Traditionally, a full characterization of a PV loop requires an invasive PV loop catheterization procedure (35) . Our objective is to utilize a noninvasive modality (i.e., echocardiography) to identify a digital twin for each patient, allowing us to simulate their individualized PV loops.Physics-based model. We utilized a lumped-parameter circuit model of cardiac hemodynamics (i.e., blood flow), also known as the Windkessel model (79) , as our underlying physical model M(θ).Lumped parameter models are based on the analogy between blood flow in arterial systems and the flow of electric current in a circuit (Figure 1 ). The model comprises interconnected compartments that are equivalent to elements of an electric circuit. Resistances represent the resistance of blood flow E K y O L p X k w T c k A w j F j B K l J Y G 5 q H z y H x Q j P u Q O S F R I 0 p 4 d p v n A 7 N m 1 a 0 p 8 C K x S 1 J D J V o D 8 8 v x Y 5 q G E C n K i Z R 9 2 0 q U m x G h G O W Q V 5 1 U Q k L o m A y h r 2 l E Q p B u N n 0 g x y d a 8 X E Q C 1 2 R w l P 1 9 0 R G Q i k n o a c 7 i x v l v F e I / 3 n 9 V A W X b s a i J F U Q 0 d m i I O V Y x b h I A / t M A F V 8 o g m h g u l b M R 0 R Q a j S m V VE D 3 V d E l O K t f X i e 9 y 4 Z z 1 W j e N 2 v t V l F H G Z 2 i M 1 R H D r p G b X S L O q i L C E r R M 3 p F b 8 a T 8 W K 8 G x / L 0 Z J R 7 F T R H x i f P 3 / 5 l K Y = < / l a t e x i t > M(✓)Step 1: Physics-Informed Pretext TaskStep 2: Physics-Guided Finetuning < l a t e x i t s h a 1 _ b a s e 6 4 = " i R q f x v l b 2 + q W l v h g l w v V 0 m B t p within the blood vessels, accounting for pressure energy losses in the system; capacitors represent the amount of stored stressed blood; diodes describe mitral and aortic valves, and inductances describe the inertial effects due to blood acceleration (83) . The blood flow within the system follows the standard Kirchoff's voltage and current laws. A patient's digital twin is represented by an electric circuit with a specific parameter instance θ, where θ corresponds to the components of the electric circuit. The patient's physiological state x corresponds to circuit currents and voltages (i.e., blood flow and pressures), which enables simulation of the patient's PV loop in any local heart structure. In this study, we adopt the five-state electric circuit model from (66) , illustrated in Figure 2 (b).I Q = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B V U m k q M u i G z d C B f u A J o T J Z N I O n U z C z E Q p I e D G X 3 H j Q h G 3 / o Q 7 / 8 Z J m 4 W 2 HC i G U 3 u W f W r Y Y 1 B V w k d k n q o E T b M 7 + c I M Z p R L j C D E k 5 s K 1 E u R k S i m J G 8 p q T S p I g P E Z D M t C U o 4 h I N 5 v + k M N j r Q Q w j I U u r u B U / T 2 R o U j K S e T r z u J E O e 8 V 4 n / e I F X h h Z t R n q S K c D x b F K Y M q h g W g c C A C o I V m 2 i C s K D 6 V o h H S C C s d G w 1 H Y I 9 / / I i 6 Z 4 2 7 L N G 8 7 Z Z b 1 2 W c V T B I T g C J 8 A G 5 6 A F r k E b d A A G j + A Z v I I 3 4 8 l 4 M d 6 N j 1 l r x S h n 9 s E f G J 8 / A 9 e Y a Q = = < / l a t e x i t > b M < l a t e x i t s h a 1 _ b a s e 6 4 = " i R q f x v l b 2 + q W l v h g l w v V 0 m B t p I Q = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B V U m k q M u i G z d C B f u A J o T J Z N I O n U z C z E Q p I e D G X 3 H j Q h G 3 / o Q 7 / 8 Z J m 4 W 2 HC i G U 3 u W f W r Y Y 1 B V w k d k n q o E T b M 7 + c I M Z p R L j C D E k 5 s K 1 E u R k S i m J G 8 p q T S p I g P E Z D M t C U o 4 h I N 5 v + k M N j r Q Q w j I U u r u B U / T 2 R o U j K S e T r z u J E O e 8 V 4 n / e I F X h h Z t R n q S K c D x b F K Y M q h g W g c C A C o I V m 2 i C s K D 6 V o h H S C C s d G w= " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B V U m k q M u i I C 4 r 2 A c 0 I U w m k 3 b o Z B J m J k o J A T f + i h s X i r j 1 J 9 z 5 N 0 7 a L L T 1 w I X D O f d y 7 z 1 + w q h U l v V t V J a W V 1 b X q u u 1 j c 2 t 7 R 1 z d 6 8 r 4 1 R g 0 s E x i 0 X f R 5 I w y k l H U c V I P x E E R T 4 j P X 9 8 V f i 9 e y I k j f m d m i T E j d C Q 0 5 B i p L T k m Q f O A w 3 I C K n M S U Y 0 9 5 w I q R F G L L v O P b N u N a w p 4 C K x S 1 I H J d q e + e U E M U 4 j w h V m S M q B b S X K z Z B Q F D O S 1 5 x U k g T h M R q S g a Y c R U S 6 2 f S H H B 5 r J Y B h L H R x B a f q 7 4 k M R V J O I l 9 3 F i f K e a 8 Q / / M G q Q o v 3 I z y J F W E 4 9 m i M G V Q x b A I B A Z U E K z Y R B O E B d W 3 Q j x C A m G l Y 6 v p E O z 5 l x d J 9 7 R h n z W a t 8 1 6 6 7 K M o w o O w R E 4 A T Y 4 B y 1 w A 9 q g A z B 4 B M / g F b w Z T 8 a L 8 W 5 8 z F o r R j m z D / 7 A + P w B + S W Y Y g = = < / l a t e x i t > b F < l a t e x i t s h a 1 _ b a s e 6 4 = " O r s d 8 U e t r 8 O D 0 z / K W A 5 V 5 7 C S 7 g M = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x I U Z d F N y 4 r 2 A e 2 Q 8 m k m T Y 0 k w x J R h i G / o U b F 4 q 4 9 W / c + T d m 2 l l o 6 4 H A 4 Z x 7 y b k n i D n T x n W / n d L a + s b m V n m 7 s r O 7 t 3 9 Q P T z q a J k o Q t t E c q l 6 A d a U M 0 H b h h l O e 7 G i O A o 4 7 Q b T 2 9 z v P l G l m R Q P J o 2 p H + G x Y C E j 2 F j p c R B h M w n C L J 0 N q z W 3 7 s 6 B V o l X k B o U a A 2 r X 4 O R J E l E h S E c a 9 3 3 3 N j 4 G V a G E U 5 n l U G i a Y z J F I 9 p 3 1 K B I 6 r 9 b J 5 4 h s 6 s M k K h V P Y J g + b q 7 4 0 M R 1 q n U W A n 8 4 R 6 2 c v F / 7 x + Y s J r P 2 M i T g w V Z P F R m H B k J M r P R y O m K D E 8 t Q Q T x W x W R C Z Y Y W J s S R V b g r d 8 8 i r p X N S 9 y 3 r j v l F r 3 h R 1 l O E E T u E c P L i C J t x B C 9 p A Q M A z v M K b o 5 0 X 5 9 3 5 W I y W n G L n G P 7 A + f w B A b u R J g = = < / l a t e x i t > y < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 m 6 i o q M A V C 3 Y o a l H + i c g Q F J d O J 4 = " > A A A B + X i c b V B N S 8 N A E N 3 U r 1 q / o h 6 9 B I v g q S Q i 6 r H o x W M F + w F N K J v N p F m 6 + W B 3 U i m h / 8 S L B 0 W 8 + k + 8 + W / c t j l o 6 4 O B x 3 s z z M z z M 8 E V 2 v a 3 U V l b 3 9 j c q m 7 X d n b 3 9 g / M w 6 O O S n P J o M 1 S k c q e T x U I n k A b O Q r o Z R J o 7 A v o + q O 7 m d 8 d g 1 Q 8 T R 5 x k o E X 0 2 H C Q 8 4 o a m l g m u 4 T D y C i W L g Y A d L p w K z b D X s O a 5 U 4 J a m T E q 2 B + e U G K c t j S J A J q l T f s T P 0 C i q R M w H T m p s r y C g b 0 S H 0 N U 1 o D M o r 5 p d P r T O t B F a Y S l 0 J W n P 1 9 0 R B Y 6 U m s a 8 7 Y 4 q R W v Z m 4 n 9 e P 8 f w x i t 4 k u U I C V s s C n N h Y W r N Y r A C L o G h m G h C m e T 6 V o t F V F K G O q y a D s F Z f n m V d C 4 a z l X j 8 u G y 3 r w t 4 6 i S E 3 J K z o l D r k m T 3 J M W a R N G x u S Z v J I 3 o z B e j H f j Y 9 F a M c q Z Y / I H x u c P I / i U A A = = < / l a t e x i t > b ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " W n V j w h 0 m H Q 0 x t 7 N G E 7 4 c D S j L S P c = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 2 m X B j c s K 9 g F N K J P p p B 0 6 m Y S Z S b G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W 5 W N z a 3 t n e p u b W / / 4 P D I P j 7 p q j i V h H Z I z G P Z D 7 C i n A n a 0 U x z 2 k 8 k x V H A a S + Y 3 h V + b 0 a l Y r F 4 1 P O E + h E e C x Y y g r W R h r b t B V h m X o T 1 J A i z p z w f 2 n W n 4 S y A 1 o l b k j q U a A / t L 2 8 U k z S i Q h O O l R q 4 T q L 9 D E v N C K d 5 z U s V T T C Z 4 j E d G C p w R J W f L Z L n 6 M I o I x T G 0 j y h 0 U L 9 v Z H h S K l 5 F J j J I q J a 9 Q r x P 2 + Q 6 r D p Z 0 w k q a a C L A + F K U c 6 R k U N a M Q k J Z r P D c F E M p M V k Q m W m G h T V s 2 U 4 K 5 + e Z 1 0 r x r u T e P 6 4 b r e a p Z 1 V O E M z u E S X L i F F t x D G z p A Y A b P 8 A p v V m a 9 W O / W x 3 K 0 Y p U 7 p / A H 1 u c P S W W U E Q = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " N E i h 1 P f b Z k v I u 9 8 F B U b V p m s N X a Y = " > A A A B 7 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 m k a I 8 F L x 4 r 2 A 9 o Q 9 l s J + 3 a z S b s T o Q S + h + 8 e F D E q / / H m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q O b R 4 L G P d D Z g B K R S 0 U K C E b q K B R Y G E T j C 5 n f u d J 9 B G x O o B p w n 4 E R s p E Q r O 0 E r t P o 4 B 2 a B c c a v u A n S d e D m p k B z N Q f m r P 4 x 5 G o F C L p k x P c 9 N 0 M + Y R s E l z E r 9 1 E D C + I S N o G e p Y h E Y P 1 t c O 6 M X V h n S M N a 2 F N K F + n s i Y 5 E x 0 y i w n R H D s V n 1 5 u J / X i / F s O 5 n Q i U p g u L L R W E q K c Z 0 / j o d C g 0 c 5 d Q S x r W w t 1 I + Z p p x t A G V b A j e 6 s v r p H 1 V 9 a 6 r t f t a p V H P 4 y i S M 3 J O L o l H b k i D 3 J E m a R F O H s k z e S V v T u y 8 O O / O x 7 K 1 4 O Q z p + Q P n M 8 f o + O P J w = = < / l a t e x i t > ✓ < lA A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I a J c F N y 4 r 2 A e 2 p W T S T B u a y Q z J H b E M / Q s 3 L h R x 6 9 + 4 8 2 / M t L P Q 1 g O B w z n 3 k n O P H 0 t h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m S j R j D d Z J C P d 8 a n h U i j e R I G S d 2 L N a e h L 3 v Y n N 5 n f f u T a i E j d 4 z T m / Z C O l A g E o 2 i l h 1 5 I c e w H 6 d N s U K 6 4 V X c O s k q 8 n F Q g R 2 N Q / u o N I 5 a E X C G T 1 J i u 5 8 b Y T 6 l G w S S f l X q J 4 T F l E z r i X U s V D b n p p / P E M 3 J m l S E J I m 2 f Q j J X f 2 + k N D R m G v p 2 M k t o l r 1 M / M / r J h j U + q l Q c Y J c s c V H Q S I J R i Q 7 n w y F 5 g z l 1 B L K t L B Z C R t T T R n a k k q 2 B G / 5 5 F X S u q h 6 V 9 X L u 8 t K v Z b X U Y Q T O I V z 8 O A a 6 n A L D W g C A w X P 8 A p v j n F e n H f n Y z F a c P K d Y / g D 5 / M H / S W R G w = = < / l a t e x i t > x LV Volume LV Pressure < l a t e x i t s h a 1 _ b a s e 6 4 = " W n V j w h 0 m H Q 0 x t 7 N G E 7 4 c D S j L S P c = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 2 m X B j cj i V h H Z I z G P Z D 7 C i n A n a 0 U x z 2 k 8 k x V H A a S + Y 3 h V + b 0 a l Y r F 4 1 P O E + h E e C x Y y g r W R h r b t B V h m X o T 1 J A i z p z w f 2 n W n 4 S y A 1 o l b k j q U a A / t L 2 8 U k z S i Q h O O l R q 4 T q L 9 D E v N C K d 5 z U s V T T C Z 4 j E d G C p w R J W f L Z L n 6 M I o I x T G 0 j y h 0 U L 9 v Z H h S K l 5 F J j J I q J a 9 Q r x P 2 + Q 6 r D p Z 0 w k q a a C L A + F K U c 6 R k U N a M Q k J Z r P D c F E M p M V k Q m W m G h T V s 2 U 4 K 5 + e Z 1 0 r x r u T e P 6 4 b r e a p Z 1 V O E M z u E S X L i F F t x D G z p A Y A b P 8 A p v V m a 9 W O / W x 3 K 0 Y p U 7 p / A H 1 u c P S W W U E Q = = < / l a t e x i t > x LV ejection fractionThe cardiac LV pressure P LV (t) and LV volume V LV (t) are related by the elastance function, E(t) = P LV (t)/(V LV (t) -V d ), which is modeled using a closed form in Eq. ( 15) in the Appendix. Evolution of x over time t is governed by a system of ODEs derived from current conservation laws (Appendix Eq. ( 20)). The model is parameterized by a vector θ corresponding to bounds on the elastance, circuit resistance, capacitance, inductance and initial states (see Appendix A.2). Given the patient-specific parameters θ i for the ODE system and elastance function governing cardiac dynamics, we have a unique solution for x i , i.e., M is injective (Appendix B).A left-ventricular assistance device (LVAD) is a blood pump that helps improve cardiac function in severely ill patients. The effect of implanting an LVAD is modeled in with the addition of one state variable describing blood flow through the device and a tuneable parameter. The impact of LVAD on blood flow can be modeled by attaching an exogenous electric circuit to M(θ) to form an augmented model M(θ) (Figure 2(b) ). Details on the mathematical model of LVAD are in Appendix A.Noninvasive data. For each patient i, we have access to an echocardiogram in the form of ultrasound video data y i and a physician labeled measurement of the LV ejection fraction xi . Echocardiography is a widely used noninvasive modality for diagnosing cardiovascular diseases; it can directly provide volumes but not pressure measurements. Our goal is to use the echocardiography clip for a patient to predict their entire PV loop through a fully noninvasive process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?[Yes] Justification: We use publicly available data and our code is already publicly available on GitHub.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
echocardiography data: 
 We test our physics-informed SSL (Med-Real2Sim) approach using two echocardiography datasets: EchoNet and CAMUS. The CAMUS dataset (44) We train a 3D-CNN implemented using Pytorch to output a subset of the model parameters θ i . The network consists of four convolutional layers, each followed by max pooling and concluded by a convolutional layer with global average pooling and two fully connected layers. We choose seven of the model parameters listed in Table 3 to be learned in our model: mitral valve resistance R M , aortic valve resistance R A , maximum elastance E max , minimum elastance E min , theoretical LV volume at zero pressure V d , start LV volume V LV (0), and heart cycle duration T C . The remainder of the parameters are fixed to literature values (see Table 3 ). We enforce restrictions on the parameters by normalizing values in our activation layer, which scales parameters using a sigmoid function to ensure parameters fall into realistic bounded ranges (as in Table 3 ). Parameters θ i are then passed to a separately trained fixed-weight feed forward neural network ϕ M to output the labeled values of xi from which we construct training loss as the mean square difference between predicted and physician-labeled V LV (t ES ) and V LV (t ED ).The network ϕ M is pretrained by generating 3,840 synthetic data points linearly sampled from realistic parameter ranges mapping parameters θ i to V LV (t ED ) and V LV (t ES ) using a numerical ODE solver (73) . We use two fully connected layers trained to minimize mean square error between predicted and outputted V LV (t ED ) and V LV (t ES ).Once predicting θ i , we can numerically solve for the complete state vector x i for each patient. The first state in x i is the patient's estimated LV volume V LV . The elastance function (Eq. ( 15)) allows us to obtain LV pressure P LV (t) from volume. We plot the two functions P LV , V LV against each other over time to obtain the PV loops. This process is deterministic, as the ODEs/elastance function are fixed by patient parameters θ i . We also use V LV (t ED ) and V LV (t ES ) to compute ejection fraction (EF) as EF = (V LV (t ED ) -V LV (t ES ))/V LV (t ED ). We compare parameters and patient PV loops across EF, a routine indicator giving information on cardiac function.Our model has good correlation between labeled and simulated end-systolic and end-diastolic volumes. Our approach, Med-Real2Sim, achieves a mean absolute error (MAE) of 6.81% for CAMUS and 5.40% for EchoNet in predicting EF (Table 1 ), comparable to a baseline that uses a 3D-CNN and fully connected layers to directly approximate F -1 using supervised learning without passing the physics-constrained layers ϕ M . Our model predicts PV loops that vary by labeled EF demonstrating that the model learns variability in patient physical state from our labels (Figure 4 (right)). Predicted parameters also vary across patient EF in our model, and the relative variation in high-and low-EF patient groups were replicated across EchoNet and CAMUS (Figure 3 ). Notably, the relationship between the parameters and patient groups by EF concur with intuited patterns. Increases in E max and E min give an increased differential between pressure and volume, which is more plausible at higher EF. R M and R A are the circuit resistances associated to the mitral valve and aortic valve, respectively. In the mathematical model, increases in these resistances must either increase pressure or decrease blood flow by Ohm's law. Decreases in either resistance clinical is considered a sign of increased hemodynamic burden, but its diagnostic value for indications including stenosis are controversial and a decrease in resistance is not necessarily predictive of reduce EF (46; 32) . In mitral regurgitation, valve resistance decreases, but EF is often increased in the beginning of the disease, which could contribute to the inverse pattern for predicted R M (52).The PV loops simulated in patient digital twins can serve as indicators for certain diseases that may not be directly labeled in the dataset. Variations in PV loops clinically enables the diagnosis of diverse cardiac abnormalities, which cannot be predicted using EF and LV volumes alone (65) . We acquired physician labels for Mitral Stenosis (MS) for a subset of the patients in the EchoNet dataset. These Using a patient's digital twin, we can simulate their counterfactual PV loops under a hypothetical intervention in-silico to estimate its effect on the patient or to optimize the treatment parameters (e.g., dosage) (56; 72) . Here, we demonstrate the feasibility of in-silico trial simulations under baseline and intervention conditions with the patient-specific digital twin M(θ i ) (Figures 4(a) ). In our setting, the lumped-parameter model of ( 66) is extended by the inclusion of one new state variable and five new parameters to model the addition of an LVAD. Tuning these new parameters and utilizing our model output allows us to simulate the effect of LVAD on an individual patient in a fully non-invasive manner.The LVAD intervention has been shown to increase EF in vivo. We demonstrate the same result in-silico using the (66) model (Figure 4 ). The average EF for the CAMUS and EchoNet populations increase by 17.6% and 18.9%, respectively, with the addition of an LVAD, consistent with reported findings (20; 18)), with patients having lower pre-LVAD ejection fractions experiencing more significant increases. Figures 4(c ) illustrates the distributional change of EF before and after LVAD implantation, indicating a significant right shift in the EF distribution following the procedure.Furthermore, the influence of rotation speed on the PV loop dynamics is a crucial factor in optimizing the therapeutic effects of LVADs. The rotation speed determines the rate at which blood is drawn from the left ventricle and ejected into the systemic circulation, directly affecting the pressure-volume relationship. Studies, such as those by Simaan et al. (66) , have explored the dynamic behavior of LVADs under varying conditions, including different initial rotation speeds. It is essential to note that the choice of rotation speed should be tailored to individual patient needs for optimal therapeutic outcomes. The impact of initial rotation speed on the PV loop, such as changes in end-diastolic and end-systolic volumes, should be carefully considered in the calibration process (Figure 4 (b)).Our physics-informed pretext task is a computationally efficient generative approximation of the dynamics of the map M. We created a second synthetic dataset of 1000 randomly sampled points to test the out of sample prediction of the physics-constrained layers ϕ M and achieve an MAE of 2.30 on EF. The distribution of the loss is uniform across plausible parameter sets suggesting that our network is not subject to poor approximations in extreme cases (Appendix Figure 9 ). In contrast, other approaches to learn the forward dynamics of differential equation models include PINNs and Neural ODEs. As previously pointed out, these models are trained to learn the dynamics of a single patient, which is incompatible with our goal of learning Eq. ( 6) with supervised learning.We compare the ability of our approach to learn the dynamics of the cardiac hemodynamics model for patients to PINN and Neural ODE methods and find it learns just as well on fewer data points (Table 2 ). Separate PINN and Neural ODE models are trained for each of N = 50 sets of patient parameters θ i . We generated using numerical methods a synthetic time series of 60 points for each of the five states in the cardiac model: D = {x i } N i=1 . From these synthetic data, we trained both a PINN and Neural ODE approach to predict x i for each of the patients computing average MAE on the 20 best results (Table 2 ). Recall that our method ϕ M is trained with volume labels (one state) only. We found that only volume labels performed poorly in the Neural ODE settings (Appendix 12). PINNs were implemented in Tensorflow (16) . Neural ODEs were implemented in Pytorch using torchdiffeq (11) .  
conclusion: 
 We present methodology, Med-Real2Sim, for non-invasive prediction of patient specific physicsbased models directly from imaging that can form the basis for further development of medical digital twins. Our experiments showcase the ability of these methods to personalize a high fidelity physics-based model from video data. We demonstrate that information beyond labels is learned in the setting of echocardiography for cardiac pressure-volume loops. Our work may be extended to improve mathematical modeling for medical or non-medical applications. Latent parameters in our experimental setting are clinically informative, and our methodology gives a non-invasive process for estimating their value. Further development of our methods could contribute new non-invasive direct ways of computing such parameters and provide the basis for in-silico digital twin studies.Limitations We highlight that our model experiments are limited in their clinical validation. In using only publicly available datasets with limited labels, we are not able to compare measurements of heart resistance or left-ventricular pressure, which would be necessary to fine-tune such a model for clinical use. While the ability to predict latent states and parameters is the core novelty of our research and approach, acquiring further data to on these latent states could improve the model substantially. This is a critical future direction for work in building digital twins with our methodology. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer:[Yes]Justification: These are both detailed in the paper's appendix and able to be referenced in the released code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our experiments do not include the calculation of uncertainty in predictions.For experiments where we present an average across patients, we show the distribution of results in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss these implications in the introduction and conclusion. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not release data nor use scraped data. While a clinical use case is the subject of our experiments, the model is not intended to be used in any clinical setting as discussed in limitations.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Data are cited. We use packages including ODE solvers which are cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have released our code along with its documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?[NA]Justification: This paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?[NA]Justification: This paper uses publicly available datasets rather than directly interacting with any human subjects. Thus, this paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The Simaan et al. ( 66) model of the cardiovascular system is characterized by the values of the circuit elements, including the capacitances, resistances, and inductance, and by the initial conditions. The model is governed by 17 parameters: eight static parameters that account for the circuit's fixed properties, four parameters that capture the elastance function, and five values describing the initial conditions (see Appendix A.2). The elastance function relates the LV volume to the LV pressureEQUATIONwhere V d is the patient specific theoretical volume of the LV at zero pressure. The elastance function is modeled in (66) asEQUATIONwith t n = t/T max , where T max = 0.2 + 0.15T c , and T c is the duration of a cardiac cycle. The evolution over time of the state of the circuit at time t is described by a 5-element vector x:EQUATIONwhere P LV , P LA , P A , P Ao are pressures of the left-ventricle, left atrium, arteries, and aorta, respectively, and Q T is total blood flow. We perform a change of variables from the (66) formulation of the model and writeEQUATIONfor ease of computation.In the model, t = 0 corresponds to end-diastole (ED), the end time of the filling phase. We assume the following constraints on the initial states• There is no restriction on x ′ 1 , the initial difference between LV volume and V d . We write x ′ 1 (0) :=start_v. • For P LA (0), the initial pressure in the left atrium, we know that the filling phase has finished, and the pressures in the left atrium and LV must be equal (63) . • The initial arterial pressure P A (0), has to be equal to the aortic pressure as at ED there is no current leaving the LV nor the aorta, so the pressures must be equal. • The initial aortic pressure P Ao (0), has to be larger than the LV pressure at end diastole but by unspecified amount. We have a second free parameter, start_pao that defines this pressure. • At ED there is not blood leaving the LV, so the initial total flow is x 5 (0) = 0. Thus, the initial state vector at ED can be writtenEQUATIONUnder these assumptions, the model's degree of freedom is constrained, and the number of independent parameters defining the system becomes 14 (8 for the static elements of the circuit, 4 capturing the elastance function, and 2 for the initial conditions)EQUATION)The evolution over time of the state vector is derived by application of Kirchoff's current law at five nodes, yielding the non-linear system of equationsEQUATIONwhere A 1 is a 5×5 time-independent matrix, and D 1 is a 5 × 2 time-independent matrix representing the activity of the diodesEQUATIONThe vector p(x) is given byEQUATION)The first element of p is the current entering the LV, and the second element of p is the current leaving the LV. These currents depend on state of the valves, which depend on pressure differentials. Equation 20, combined with a set of initial conditions x(0), provides the evolution over time of the volume of the LV V LV and the pressures in the system. The evolution of V LV (t) and P LV (t) gives patient pressure-volume loops.We simulate the addition of a left ventricular assist device (LVAD) to the cardiac system. One end of the device is connected between the left ventricle and the aorta, and the other end is connected between the left ventricle and the left atrium. Many different LVADs have been developed; we take here the case of a rotary-controlled LVAD as described in (12; 66) .The presence of the LVAD induces the addition of a new state variable, x 6 , representing the blood flow through the device. In addition, the LVAD representation in the electric circuit is described by parameters R 0 , R i , P , α, L i , L 0 , β 0 , β 1 , β 2 , -describing compliances, static resistances and a time-varying resistances-and by the pump rotational speed, ω(t), which is a function of time.The pump speed can be defined beforehand (e.g. a linear increasing function), or it can be updated in real-time based on the state of the system. A more detailed description of the parameters and equations of the LVAD can be found elsewhere (12; 66) .By application of circuit analysis laws, new equations of the circuit describing the cardiovascular-LVAD system are givenEQUATION) for new state vector y, containing the first five variables as in 18 in addition to the blood flow through the LVAD, x 6 (t). The new matrices of the system areEQUATION) R k is a time-dependent resistance, only allowing current flow through the LVAD when the pressure in the left ventricle is below a threshold noted P . It is given byEQUATIONThe vector p(x) is as defined in 22, and B(t) is given byEQUATION)A. H = β0x6 + β1 dx 6 dt + β2ω 2Table 3 : Parameters of the lumped-parameter hydraulic analogy model of the left-ventricle.First, this function is well defined: given θ, which includes parameters describing the initial system state, there is a unique ODE solution x. Using Picard-Lindelöf, it is sufficient for A 1 x + D 1 p(x) to be continuous in t and Lipschitz continuous in x. As x describes pressures and flows that cannot be discontinuous in t, the first condition is satisfied. As A 1 x + D 1 p(x) is linear in each of the four cases defined by the pairwise possibilities for the rows of p(x) to be either positive or zero, we can take the maximum needed Lipschitz constant amongst these four possibilities to see the entire equation is Lipschitz continuous.Conversely, given complete knowledge of x, we can find the twelve static model parameters in θ uniquely. We assume that complete knowledge of x includes knowledge of the five state variables (V LV -V d , P LA , P A , P Ao , Q T ) over time in addition to the complete knowledge of P LV , V LV . The difference between V LV and the first state variable gives V d . The ratio P LV /V LV -V d gives E(t).The maximum and minimum values of this function give E max and E min , respectively. The value of T c is the length of a period of E(t). This gives the four parameters describing elastance.Applying the Laplace transform L(f ) = ∞ 0 e -st f (t)dt to Equation 20, we haveEQUATION)The vector L(p(x)) is constant. For any fixed s, L(x)(s) is also a constant. Thus, each s gives a linear system of five equations in the eight unknown static parameters governing the Simaan et al. (66) model. Identifying these eight parameters uniquely simply requires choosing enough values of s to solve each row as its own linear system. This shows that M is injective.Generally, inverse problems are the task of determine inputs θ ∈ Θ from measurements y ∈ Y in the set up F(θ) = y with forward operator F : Θ → Y. Inverse problems are typically ill-posed, meaning that the forward model F may be non-injective or that small perturbations of measurements y (i.e. noise) cause large change in the corresponding inputs θ. Often, a variational approach is taken to mitigate these challenges, meaning the problem is reformulated into the task of minimizing over the possible solution set Θ the sum of a data fidelity term D to represent fitting the model F in addition to a term incorporating solution criteria R at some level α:EQUATIONFor example, D might be ||F(θ) -y|| to encourage model fitting and R might be ||θ|| to encourage input sparsity. This optimization task can been solved using analytical techniques to find θ directly or using generative learning techniques to approximate F -1 to find θ.We also seek to solve an ill-posed inverse problem as the forward model F is non-injective. There are different patient states that could look nearly identical on imaging. In our setting, as we further break down this ill-posed task of inverting F as that of inverting F = K • M for components M : Θ → X and K : X → Y, we seek to solve an ill-posed inverse problem with partially known forward model. Other strategies for inverse problem with partially known forward model include adversarial methods such as CycleGAN for unpaired signals and measurements and AmbientGAN for training from measurements alone with parametric forward model. CycleGAN simultaneously learns the forward and inverse process by minimizes the expected value of the difference between ground truth inputs and outputs and with forward and inverse model composition outputs (86) . AmbientGAN learns the inverse model by discriminating between the distribution of the measurements and distribution of the forward model applied to the modeled inputs (5) . For settings with unlabeled data, (19) pioneers Untrained Physically Informed Neural Networks (UPINNs), training a CNN to reconstruct images by reducing the training error between the original measurements and outputs passed through a known forward model.These existent set ups differ from our task as our partially known model is really the composition of a known and unknown model. We take a generative inverse problem approach and develop a model for F -1 , which we compare to other generative solutions in Table 4 across the following two taxonomies: known versus unknown forward model A (row) and paired versus unpaired input-output data (column). This taxonomy mirrors that in (51) .Efforts to incorporate knowledge into neural network models to improve training and accuracy include techniques to leverage physical models represented by differential equations (75) . In big data settings, physical laws may be learned during modeling. In smaller data settings, incorporating physics into models can improve performance (75) . This is especially pertinent to personalized medicine where data is inherently limited. Physics-informed neural networks (PINNs) typically have the goal of either finding solutions (forward) or system parameters (inverse) to systems of differential equations Forward task (learn x(t))Inverse task (learn θ) Traditional PINNs with f as regularization term (39) Chen et al. (10) proposed the use of differential equation solvers for solving neural networks upon noticing the similarity between ResNet gradient descent and Euler's method. Viewing neural networks as ODEs means forward propagation is the system solution. It also means that, in contrast, to ResNet architectures, the network does not have a fixed number of layers. Neural ODEs are typically used to model time series data to replace ResNets. The set up is as follows.Let ODESolver represent a numerical ordinary differential equation (ODE) solver. A fully connected neural network f will represent the ODE the network learns to solve so that a forward pass looks like ODESolver(f, t 0 , x 0 , T )for initial condition x(t 0 ) = x 0 and prediction time series T . In this set up, f can represent an ODE that we seek to approximate, analogous to our setting. Neural ODEs are trained by back propagating through the numerical ODE solver. Methods to reduce the computational cost of this backpropagation include the adjoint state variable methods in (10) and augmenting the system to a higher dimensional space (21) .Our fixed pre-training task for leveraging the physical model shares similarities with Neural ODEs in that we train a network to learn to solve an ODE. Our aim differs as a Neural ODE can learn the dynamics for a single patient. The ODE f will be specific to a single patient parameter set θ, but we seek to learn a map for a patient population.The Simaan et al. (66) model has five state variables which are determined by solving a system of ordinary differential equations (ODEs). We derive five ODEs from basic circuit current laws. We obtain equations for the derivative of each of the five state variables. RecallEQUATIONand that in our case, current is flow and voltage is pressure.The following basic electric circuit facts are sufficient for the derivation.• For resistors: current = voltage /resistance, i.e. I = V /R.• For capacitors: current = compliance * derivative of voltage with respect to time, i.e. I = C dV dt .• For inductors: voltage = inductance * derivative of current wrt t, i.e. V = L dI dt . • Kirchhoff's laws: current must be conserved at any node and voltage sums to zero around any loop.• I = dQ dt current is the change in charge Q.The first row of the ODE system is given by conserving the current I at the red node in Figure 5 as-I 3 = I 1 -I 2whereI 3 = (C(t)x 1 ) ′ = x ′ 1 + C ′ (t) C(t) x 1 , I 1 = max(x 2 -x 1 , 0)/R Mas the voltage flowing across the resistor would be the difference between the voltage at the two capacitors (using Kirchhoff's law) and the diode prevents flow in the negative direction, and I 2 = max(x 1x 4 , 0)/R A . Combining and rearranging, we get The second row is given as the following by conserving current at the yellow circled node in Figure 5 (x ′ 1 = -C ′ (t)/C(t)x 1 + max(x 2 -x 1 , 0)/(C(t)R M ) -max(x 1 -x 4 , 0)/(C(t)R A ).x 3 -x 2 )/R S = C R x ′ 2 + max(x 2 -x 1 , 0)/R M x ′ 2 = (x 3 -x 2 )/(R S C R ) -max(x 2 -x 1 , 0)/(R M C R ).The third row is given asC S x ′ 3 = (x 2 -x 3 )/R S + x 5 x ′ 3 = (x 2 -x 3)/(R S C S ) + x 5 /C S by conserving current at the yellow node in Figure 5 .The fourth row is given asC A x ′ 4 = -x 5 + max(x 1 -x 4 , 0)/R A x ′ 4 = -x 5 /C A + max(x 1 -x 4 , 0)/(R A C A )by conserving current at the green node in Figure 5 .The fifth row is given usingL dI dt = V as L S x ′ 5 = (x 4 -x 3 ) + max(x 1 -x 4 , 0)/R A x ′ 5 = (x 4 -x 3 )/L S + max(x 1 -x 4 , 0)/(R A L S). This relation is written using the relation for inductors above so the left hand of the first line is L dI dt and the right hand side is voltage. In total, after performing a change of variables of x 1 to x ′ 1 := x 1 /E(t) these five equations give us Equation 20.We sought to find a reduced order equivalent circuit to improve model performance but were unsuccessful in finding an identifiable system. We began by ignoring diodes in our circuit to use linear equivalence, as then our circuit would have only linear components (capcitors, inductors, resistors), and we can use the follow result. Theorem D.1 (Norton). Every linear circuit is equivalent to one with a single current source and single resistor in parallel with a load of interest.R N O I N O C(t)For our case this gives us the circuit in Figure 6 , as we are interested in the capacitor representing pressure change in the left ventricle.To identify this circuit, we wrote the variables of this new circuit in terms of the state variables in the (66) model. To calculate the resistance of the equivalent circuits, we start by shorting all voltage sources (including capacitors) and opening all inductors. This gives us the circuit in Figure 7 . As this is in fact the resistors R S and R A in series and R M in parallel, using current divider rules, the Norton resistance isR M R A C(t) R C R SEQUATION)With the addition of diodes, since ours are ideal, when current flow is in the diode's forward direction, we replace the diode with a short. In the reverse direction, we replace the diode with an open, which would vary the calculated resistance to include and exclude resistances.We would want to leverage the reduced circuit architecture to solve for P LV (t). We know that the current through the load capacitor is I cap = (C(t)P LV (t)) ′ using basic capacitor circuit relations. The Norton equivalent circuit gives a node to solve using Kirchoff's law,EQUATION)where I R N O is the current flowing through the Norton resistor. Thus, we writeEQUATIONEQUATIONgiving two unknown state variables I R N O (t), P LV (t) and one ODE, which is insufficient to determine the circuit.The EchoNet model, achieving a Mean Absolute Error (MAE) of 5.40%, was trained for 3 hours over 13 epochs with a batch size of 100 and a learning rate of 0.001. In comparison, the CAMUS model, which reached an MAE of 6.81%, required 2 hours of training over 110 epochs with a batch size of 50 and a learning rate of 0.005.In our experiments, we conducted a comprehensive hyperparameter search to ensure optimal performance. We employed a grid search approach to systematically explore a range of values for key hyperparameters. Specifically, we evaluated learning rates of 0.001, 0.005, and 0.0005, batch sizes of 50 and 100, and the number of epochs set to 300 and 500. We used the adam optimizer for training.Models were trained on CPUs, each node featuring 24 cores (12 physical cores with hyper-threading) and 128 GB of RAM. The distribution of the loss is uniform across plausible parameter sets suggesting that our network is not subject to poor approximations in extreme cases. Justification: We have a limitation section pointing out the need for additional clinical validation of models built from these methods. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This manuscript proposes a novel method for creating patient-specific digital twins focused on cardiac hemodynamics. The general idea is how to infer from a non-invasive measure an invasive measure. The domain specific task aims to predict the physiological signal such as heart pumping blood from the recording of an echocardiogram image. The novelty of the method is to solve the associated inverse problem specifically for that patient, so that personalized predictions can be performed. The proposed method uses a two-steps Self Supervised Learning (SSL) approach that can be summarized by a step of pre-training and a step of fine tuning. A neural network is pretrained to learn from the data of a differentiable simulator the parameters of cardiac process. Then, another model is trained to reconstruct physiological measurements from non-invasive data while being constrained by physical equations learned during pretraining.

The rebuttal allowed the clarification of several aspect and the Reviewers agreed that the discussion was fruitful for a better understanding of the proposed method. No major concerns affect the soundness of this work after the rebuttal.

The controversial aspect of this work remains how should be perceived from the scientific community. The Authors believe that their contribution is general and the application to cardiac hemodynamics has only the purpose of an explanatory example. The Reviewers complain about this claim at least for two reasons: (i) the assumptions for the generalization are not clearly formulated, (ii) the successful application of this approach to other use cases is purely speculative and without empirical evidence. 
The proposed framework assumes at least a couple of premises: a simulator that encodes the relationship between a target variable x and an intermediate variable w, a collection of data where the annotation between an input variable y and the variable w is available (where x would be the invasive measure and y the non invasive measure). Even though this pattern can be recognized in several applications, the implementation is use case dependent.
For this reason this work could be considered for poster presentation.

Reviews: review: paper_summary: paper_summary: The paper proposes a method to identify parameters for digital twin models of patients using non-invasive health data, eliminating the need for invasive procedures. This method focuses on scenarios like cardiac hemodynamics, where traditionally invasive measurements (e.g., through catheterization) can be predicted using non-invasive data (e.g., echocardiograms). The novelty of the method is to solve the associated inverse problem specifically for that patient, so that personalized predictions can be performed.

The proposed method uses a two-step SSL approach that structurally resembles pretraining and finetuning in SSL. First,  a neural network is pretrained on synthetic data to learn the forward dynamics of a physics-based model. Then the pretrained model is then used to train another network on actual non-invasive patient data to predict physical parameters.
Application to Cardiac Hemodynamics:

The paper illustrates how to apply the above method for cardiac hemodynamics using echocardiography. This allows the prediction of patient-specific pressure-volume (PV) loops using non-invasive echocardiogram videos.
strength_weakness: strength_weakness: The authors focus only on a very specific medical use-case, they don't try to generalize their method to more cases. In the introduction the authors illustrate this as a generic approach, therefore I was surprised to not find an attempt to support their vision with more application examples. Without the demonstration that this method can have a broader use I don't think this paper is suitable for presentation at this conference. Also, the authors don't run convincing ablations to demonstrate that their approach is sounding.
questions: questions: Have you experimented with this method beyond Cardiovascular Hemodynamics?
You implement a 3D-CNN - how did you arrive at such architecture? have you run ablations?
limitations: limitations: The authors mention the limited clinical validation. This is certainly one limitation to take in account. I would consider also the lack of generalization to different clinical procedures.
paper_summary: paper_summary: I have read this manuscript during ICML review. It looks the same so I copied my previous review.

The authors presented a method to infer the physical parameters θ of physiological process (heart pumping blood) from noninvasive observation y (the echo image). The mapping from y to θ cannot be directly learned due to the lack of paired data. Instead, they find that an intermediate variable x_bar (the EF) can be annotated by experts (x_bar=g(y)) which can also be calculated based on θ (x_bar=m(M(θ))). Thus the observable pair (y, x_bar) provides the supervision for learning θ = F_inv(y), through the relation x_bar=m(M(F_inv(y))) where m is rule-based and M is a solution to ODE. They first train a surrogate network to approximate M(θ) through synthetic simulation, followed by learning the parameter of F_inv on observations {(y, x_bar)}.
strength_weakness: strength_weakness: 1.The authors compared their methods in predicting EF from echocardiogram with supervised 3DCNN but did not outperform them in MAE. It should be noted that the EF is calculated on ED and ES segmentation and supervised segmentation network is expected to perform even better.
2.Lack of validation. The authors performed validation by comparing EF derived from the physical parameters to the observed EF. But this does not guarantee the correctness of their physical parameters. In fact, arbitrary intermediate variables can be defined and could also lead to the comparable prediction of EF.
3.Training the surrogate model of x=M(θ) need to generated synthetic samples while this is domain-dependant. Whether the learned mapping fits extreme or shifted situations is unknown, lacking of uncertainty analsysis.
questions: questions: 1.Could the authors provide more solid validation of their inferred physical parameters?
2.Could the authors provide the uncertainty/robustness/generalisation ability of the surrogate model?
limitations: limitations: My biggest concern is that the method lacks ground-truth validation, at least some samples.
paper_summary: paper_summary: This paper introduces a novel methodology for identifying patient-specific digital twins using noninvasive medical imaging, particularly focusing on cardiac hemodynamics. By leveraging a physics-informed self-supervised learning approach, the research addresses the challenge of modeling digital twins without invasive data collection. The process involves pretraining a neural network to simulate physiological processes and then fine-tuning it to reconstruct physiological measurements from noninvasive modalities, constrained by the learned physical equations. This framework allows for the simulation of patient-specific physiological parameters and the potential for conducting in-silico clinical trials and unsupervised disease detection.
strength_weakness: strength_weakness: 1. Insufficient performance comparison. The paper only compares PINN and Neural ODE methods. There are many variations of PINNs methods which outperform the original one. The paper should choose more solid baselines.

2. No ablation study. The paper proposed a two-stage training strategy, but it didn't show why it is necessary.
questions: questions: 1. How do the results of the model change if the estimated parameters are dynamic rather than constant? Considering the medical context where a patient's condition is continually changing, assuming constant parameters seems unrealistic. How might this affect the reliability and accuracy of the model used in such scenarios?

2. Can you explain why blood flow is often modeled using an electrical circuit analogy, and why it adheres to Kirchhoff's laws of voltage and current?
limitations: limitations: The proposed method requires a pretty strong assumption of the PDEs dynamics. I am a little bit worried about the training efficiency of this method. Can it be used as a real diagnostic tool? Additionally, I am intrigued by the potential of this method to be applicable across a broader range of medical disciplines.
paper_summary: paper_summary: This paper proposes a novel method for creating patient-specific digital twins using non-invasive patient health data. The authors introduce a physics-informed self-supervised learning (SSL) algorithm that pretrains a neural network on learning a differentiable simulator of the cardiac process. Then, another model is trained to reconstruct physiological measurements from non-invasive data while being constrained by physical equations learned during pretraining. The method is applied to identify digital twins of cardiac hemodynamics using echocardiogram videos, showing good results in unsupervised disease detection and in-silico clinical trials.
strength_weakness: strength_weakness: * Simplifications/assumptions in the Windkessel and LVAD models might not fully capture the complexity of the heart dynamics.
* The results might be sensitive to low quality non-intrusive data, which can affect the global accuracy of the method.
* It is not clear the demographic diversity of the echocardiography dataset. Thus, the model might not generalize well across all the segments of the population.
questions: questions: * Lines 251-255, Table 3: Which criteria was used to select the learnable and the fixed parameters of the model? Are these selected by using some kind of sensitivity analysis based on the state-space matrices in Eqs. 21 and 24?
* Line 708: Appendix C1 already addresses the ill-possedness of the inverse problem. That means that the trained model could potentially assign the same digital twin to two different patients due to the similarity of their echocardiograms. Have the authors found any difficulty in this regard? 
* Have the authors considered using an easier and more accessible nonintrusive techniques such as electrocardiograms? Would the use of several modalities of non-intrusive data for the same patient increase the performance of the model?

Minor comments:
* Line 227: "tune-able" might refer to "tunable".
* Line 252: Incorrect reference, Table A.2. might refer to Table 3.
* Line 814: Incorrect reference, Figure D.2. might refer to Figure 7.
* Figure 7: The figure model seems to be incomplete/offset.
* Equation 35: A parenthesis is missing after $R_{NO}$.

Final comment: The paper is well structured, the results are promising and the methodology has moderate impact on the field. However, it's still unclear if this model could handle the variability and complexity of human physiology by only non-invasive measurements. Based on the comments above, I reccomend a weak accept.
limitations: limitations: Limitations were addressed appropriately.

Comments: comment: Thank you very much! Your reply almost addresses my concerns.
I have read the authors reply and all the other reviews. I appreciate the effort of the authors in providing more details and explanation, they helped me understand the paper and their motivations better. I slightly increased my score. However my concern on the generalizability of the method and the missed opportunity to demonstrate it remains. (and I can see it's a shared concern among the reviewers).
I thank the authors for the rebuttal, I have no more concerns.
Thank you for your rebuttal. My concerns have been addressed to some degree. I have increased my rating.