Robust Hybrid Learning With Expert Augmentation
Authors: Antoine Wehenkel
Jens Behrmann
Hsiang Hsu
Guillermo Sapiro
Gilles Louppe
Joern-Henrik Jacobsen
Venue: Accepted by TMLR ()

Abstract: Hybrid modelling reduces the misspecification of expert models by combining them with machine learning (ML) components learned from data. Similarly to many ML algorithms, hybrid model performance guarantees are limited to the training distribution. Leveraging the insight that the expert model is usually valid even outside the training domain, we overcome this limitation by introducing a hybrid data augmentation strategy termed \textit{expert augmentation}. Based on a probabilistic formalization of hybrid modelling, we demonstrate that expert augmentation, which can be incorporated into existing hybrid systems, improves generalization. We empirically validate the expert augmentation on three controlled experiments modelling dynamical systems with ordinary and partial differential equations. Finally, we assess the potential real-world applicability of expert augmentation on a dataset of a real double pendulum.

Published in Transactions on Machine Learning Research (02/2023)
Robust Hybrid Learning With Expert Augmentation
Antoine Wchenkel
auchenkel@@pple..om
Apple
Jens Behrmann
i behrmmennn@aplee.com
Apple
Hsiang Hsu
hasianghsu@gg.hrrrordd.c
Horvard
Guillermo Sapiro Apple
gsapiro@apple.com com
Gilles Louppe University of Liège
goxppee
Jörn-Henrik Jacobsen Apple
@ajucclben@@appl...o com
Reviewed OpenReview: https: // openreview. net/ porm? id= de4
Abstract
Hybrid modelling reduces the misspecification of expert models by combining them with machine learning (ML) components learned from data. Similarly to many ML algorithms, hybrid model performance guarantees are limited to the training distribution. Leveraging the insight that the expert model is usually valid even outside the training domain, we overcome this limitation by introducing hybrid data augmeentttion strategy termed expert augmentation. Based on probabilistic formalization of hybrid modelling, we demonstrate that expert augmentation, which can be incorporated into existing hybrid systems, improves generalization. We empirically validate the expert augmentation ments modelling dynamical systems with ordinary and partial differential equations. Finally, we assess the potential real-world applicability of expert augmentation on a dataset of real double pendulum.
three controlled experi-
Train distribution
disifoution
Test distribution
Ground truth (ouns))
APIYUUTT+
APHINIITY
Time (sccond)
Figure 1: APHYNITY, existing hybrid modelling strategy, is unable to predict accurately the dynamic of diffusion reaction for shifted test distribution, although predicts configurations that follow the training distribution. APHYNITY+, the same model fine-tuned with expert augmentation, generalizes to shifted distributions as expected from the validity the underlying physics.

---

Published in Transactions on Machine Learning Research (02/2023)
1 Introduction
Generalizing to unseen data is crucial to make model applicable in the real world. When training and test data are independently and identically distributed (IID), we assess the model generalization on held-out. subset of the training data. Unfortumately, the training and test scenarios do not entirely overlap in practice. This observation has motivated many recent research efforts to focus on the robustness of ML models (Gulrajani & Pazez-Paz,, 2020; Geirhos et al., 2020; Koh et al., 2021). Common strategies can be broadly grouped into two categories. The first class of methods aims to align properties of the model (e.g., invariance, equivariance or monotonicity) with expertise in the problem of interest (Cubuk et al., 2019; Mahmood et al., 2021; Keriven & Peyré, 2019; Silver et al., 2017). The second category is data-focused (Sagawa et al., 2019; Arjovsky et al., 2019; Krueger et al., 2021; Creager et al., 2021), and leverages variations present in the training data, c.g., some methods minimize the worst-casc sub-group performance, to achieve robustness.
The data-oriented methods, which include Group-DRO (Sagawa et al., 2019) and Invariant Risk Minimiza- tion (Arjovsky et al., 2019, 1RM), present the advantage to specify the invariances implicitly via domains or environments. However, these methods rely on variations in the training data, which may be insufficient when the problem is too complex, or the variations of interest are absent from the training set. On the other hand, methods based on domain-specific expertise do not suffer from such limitations. Embedding expertise into a model can be done via architectural inductive biases (LeCun et al., 1995; Xu et al., 2019), data augmentation learning objective that enforces established symmetries of the problem (Cranmer
(Cubuk et al., 2019), et al., 2020). For example, simple data augmentation techniques combined with convolutions lead to excellent. performance on natural image problems (Cubuk et al., 2019). Another natural approach to embedding expertise in ML models, and the one studied in this paper, is called hybrid learning, or sometimes gray-box modelling (Willard et al., 2022). This framework combines an expert model (e.g., from first principle physics) with a learned component that improves the expert model so that the combination better fits real-world data. In hybrid learning, the expert model plays a central role and is supposed to provide simple and well-grounded (parametric) description of the process considered.
In recent works (Yin et al., 2021; Takcishi & Kalousis, 2021; Qian et al., 2021; Mehta et al., 2021; Lei & Mirams, 2021; Reichstein et al., 2019), hybrid learning demonstrated success in complementing partial physical models and improving the inference of the corresponding parameters. We observe that current hybrid learning algorithms are sub-optimal in the amortized inference setting- when we aim to build hybrid models that are valid for various test configurations. Contrary to the common belief that hybrid learning achieves better generalization than black box ML. models, we argue and demonstrate that hybrid learning algorithms do not yet meet their promise regarding robustness in amortized settings. Although hybrid learning achieves strong performance on IID test distributions by exploiting the inductive bias of the expert models, their performance collapses when the test domain is not included in the training domain. This is unsatiffcttrry as the expert model is typically well-defined far outside the training distribution.
We introduce expert augmentations for training augmented hybrid models (AHMs), a procedure that exlends the range of validity of hybrid models and improves generalization, as pictured by Figure 1. Our contribution is to first formalise the hybrid learning problem as: 1) Learning probabilistic model partially defined by the expert model; 2) Performing inference over this probabilistic hybrid model. In this context, we show that hybrid learning is vulnerable to ristrinution shifts for which the expert model is well defined (see Figure 1, bottom row). Motivated by our analysis, we propose to fine-tune the hybrid model on an expert-augmented dataset that includes distribution shifts (sce results of augmentation in Figure 1, middle row). These expert augmentations only rely on the hybrid model itself,, leveraging that the expert model is also well-defined outside of the training distribution. Our experiments on various controlled problems demonstrate that AHMs improve the generalization capabilities of state-of-the-art hybrid learning algorithms on synthetic and real-world data in the amortize setting.
2 Hybrid learning
We formalize hybrid learning with the probabilistic model depicted in Figure 2, and later rely on this formalism to show the benefits of the proposed proposed expert augmentation. In this Bayesian network,
2

---

Published in Transactions on Machine Learning Research (02/2023)
capital letters denote random variables (e.g., Y) and, we use calligraphic letters for the domain of the corresponding realization (e.g.; y E 3r). In our formalism, the expert model is conditional density p(ye (z, ze) that describes the distribution of the expert response Ye to an imput a together with a parametric description of the system Ze, denoting expert parameters. We augment the expert model with the interaction model which is a conditional distribution plylæ, Me, =a) that describes the distribution of the observation Y given the imput 2, the expert model response Ye, and a parametric description of the interaction model Za-
Our goal is to create robust predictive model plyla, (Io, Hol) of the random variable Y, given the in- put x together with independent observations (Xo, No) of the same system, where the subscript denotes an observed quantity. As a concrete example, we consider predicting the evolution of a damped pen- dulum (described in Scction 4.1) given its initial angle and speed (xr [0.0] and sequence of ob- servations of the same pendulum. The expert model we assume is able to describe a frictionless pendu- lum whose dynamic is only characterized by one parameter 2e := wo, denoting its fundamental fre- quency. The expert model is misspecified; it does not model the friction with a second parameter 2g := a, the damping factor. In this problem, (oo: Vo) and (x, y) are IID realizations of the same pendulum which corresponds, in general lerms, to samples from p(r, ylea, ze) for some fixed but unknown values of za and ze. The expert variables ze (e.g., wb) together with &a (e.g, a) should accurately describe the system that produces Y (e.g., the evolution of the pendulum's 's angle and speed along time) from X (e.g., the initial pendulum's state). In our setting we assume that we are given pair (z) (e.g., past observations) from which we can accurately infer the state of the system (e) as described by the interaction and expert models, and then prodict the distribution of Y for a given input (e.g., forecasting future observations) to the same system. Provided all probability distributions in Figure 2 are known, the Bayes optimal hybrid predictor PB is
Expert model
Ze
X
Za
Ye
Y
hybrid probabilistic model which describes the relationship between given input X and the output Y for a configuration of the system as defined by the la- tent variables Z, and Z.. The prescribed expert model defines the conditional density p(ye) že, x), where Y. is an approximation of Y. Hybrid learning aims at learning the conditional distribution p(ylza Za. ye, x).
Figure 2:
Pa ( x (Io. = E Epiz..e (Ronw))) [ply|, Ras 2a: ze)]
(1)
shown in Appendix F.
In the amortized setting, we aim to learn model of both the predictive model p(y|e, Ra, ze) and of the posterior over the parameters p(ža, ze (xo, vo)). We will see that existing hybrid learning algorithms neglect the importance of building a robust encoder p(z,, =e) to make predictions in out-of-distribution (OOD) settings. In particular, we note that the interaction between &e and y is essentially defined by the expert model. Thus it should be possible, and preferable, to learn a predictive model of Y whose performance guarantees are as independent as possible from the training distribution of the expert variables ze. Indeed, demonstrate below
the validity of the expert model often go beyond the seen training examples. However, that existing hybrid learning algorithuns' performance collapses when the distribution of ze shifts. The expert augmentation introduced in this paper improves the robustness of hybrid models to such shifts.
2.1 Hybrid generative modelling
As common in the hybrid learning literature, consider expert models that are deterministic although our discussion clcarly extends to stochastic expert models. The expert model describes the system as function Je : X' x Z. + Ye that computes the response Ye to an input x, parameterized by expert variables z,- The goal of hybrid modelling is to augment the expert model with a component learned from data as depicted in Figure 2. Formally, given a dataset D = {(x(0,, ())) of N IID samples, we aim to learn the interaction model (2 Ve. 2a) that fits the data well but is close to the expert model. Defining closeness is hard and is application-depeneent modelling choice beyond the scope of this paper. However, common metrics include
3

---

Published in Transactions on Machine Learning Research (02/2023)
L2-distance between expert and hybrid outputs Kullback-Leibler (KL) divergence between the marginal distributions of Y and Ye-
Learning a model that is close to the expert model and fits the training data well is a hard problem. However, the APHYNITY algorithm (Yin et al., 2021) and the Hybrid- (Takeishi & Kalousis, 2021, HVAE) are two recent approaches that offer promising solutions to this problem. We now briefly describe how these methods approximate the Bayes optimal predictor of Equation (1). Our augmentation strategy is compatible (and effective) with both approaches.
Yin et al. (2021) consider hybrid learning for augmenting expert models defined as ordinary differential equation (ODB). They consider an additive hybrid model that should allow a perfect fitling of the data. In probabilistic terms, this assumption is equivalent to assuming the conditional distribution po (y 22, Me. 2a) is a Dirac distribution. Formally, they solve the following optimization problem
APHYNITY.
dy:
min Pe,Pa
Fall
s.t. v(r,y) E D, Vt, dt
= FUl, 2e) + Fa(1))
with yo := Z,
(2)
where || is a norm operator on the function space, Fa : Vi + Vr is a learned function, F. 3/ x Ze + Vo defines the expert model and DD is a dataset of initial states x := yo and sequences y E E : (Di)*; where k is the number of observed timesteps. APHYNITY solves this problem with Lagrangian optimization and Neural ODFs (Chen et al., 2018) to compute derivatives. In the context of ODFs, the random variable X is the initial state of the system at to and Y is the observed sequence of k states between to and f1
This formulation only considers learning missing dynamic for one realization of the system described by Figure 2, for a single ze (and za). However, we are interested in learning a hybrid model that works for the full set of systems described by Figure 2. As suggested in Yin et al. (2021), we use an encoder network (x())): :x)))x(() xy + Zu x Ze that corresponds to Dirac distribution located at gv as the approximate posterior (2a, Ze (), v)- The interaction model is product of Dirac distributions whose locations correspond to the solution of the ODE
dy:
Fa ze) + Falye, Za; 0), yo
x.
(3)
dt
Hence the corresponding approximate Bayes predictor replaces the parameters (za: 2e) in Equation (3) with the prediction of go and predicts a product of Dirac distributions.
Hybrid-VAE (HVAE). In contrast to APHYNITY, the hybrid-VAF, proposed by Takeishi & Kalousis (2021) is not limited to additive interactions between the expert model and the ML model, nor to ODEs. Instead, their goal is to learn the generative model described by Figure 2. They achieve this with variational auto-encoder (VAE) where the decoder specifically follows Figure 2. Similarly to the amortized APHYNITY model, the encoder (x y) prodicts a postcrior distribution over za and 2e, and the model is trained with the classical Evidence Lower Bound on the likelihood (ELBO). Takeishi & Kalousis (2021) observe that relying only on an architectural inductive bias and maximum likelihood training is not enough to ground the generative model to the expert equations. They propose to add three regularizers Rppc, RDA.1, and Rpa,2 that encourage the generative model to rely on the expert model. The final objective is
max Ep ELBO((#, y): sb 0)] + aRppC + +BRDA1 + 7RDA22 0.49
(4)
The first regularizer, RppC, encourages the marginal distribution of samples generated by the complete model to be close to the marginal distribution that would be only generated by the physical model. The two other regularizers specifically require the encoder network for ze to be made of two sub-nelworks. The first network filters the observations to keep only what can be generated by the expert model alone; it predicts 2, and remove its effect from y to predict a filtered version of it. The sccond network maps the filtered observations to the posterior distribution over 2.. RDA,1 penalizes the objective the observations generated by the expert model are not close to the filtered observations. Finally, RDA_ relies on data augmentation
4

---

Published in Transactions on Machine Learning Research (02/2023)
with the expert model to enforce that the second sub-network correctly identifies the expert variables if the observations were correctly filtered by the first encoder. We refer the reader to Appendix B and Takeishi & Kalousis (2021) for more details on HVAE. For HVAE, the approximate predictor takes the form described by Equation (1) where p(za, ze Fel( is approximated by the encoder qu (2,, 2e|2, y) and p(y]a, Pas ze) by the learned hybrid generative model.
pize x, y) 2 qu (zel v)
Ze
Zu
Zu
Ze
Ω
p(z, B12c) bnt a. ye. za)]
= (p(z)))((a (r, (ex))p(())z Ve, 2a)]
Figure 3: Visualization of the distribution shifts considered in this work. The train support Q of (z, y) results from (ea,2e) Ze x Ze. The test supports (in red) denoted with tilde symbols 2g for ze and ft for (z, y) The augmented support Q (in green) includes both train and test scenarios and corresponds to (zo, =.) E Z. x Z.. The outer violet domain that includes 5 depicts one of our experiment in which the domain of & is also shifted. Hybrid modelling algorithms alone may learn a mapping Po 2. + 52 but augmentation is necessary to learn the inverse mapping de + Ze.
3 Robust hybrid learning
We now formalize our definition out of distribution (OOD) and robustness. In general, a test scenario is OOD if the joint test. distribution p(x, y) different. from the training distribution p(a,y), that d(p, p) > for any properly defined divergence or distance d between distributions. In the following, we reduce our discussion to sub-class of distribution shifts for which the marginal train and test distributions over & may be different, d(p(ze), ū(=x)) > O, but the marginals of za and are constant. As a consequence, the joint distribution of (I, v) pairs is also shifted. Formally, the training and test distributions are respectively defined as
p(z, v) = (odd)) [p(x)p(ul>, Ver =a)] p(x, v) == Dpte..p( [p(x)p((]z) Mes za)]
In this context, we demonstrate, theoretically and empirically, that classical hybrid models fail. To address this failure, we introduce augmented hybrid models and show that, under some assumptions, they achieve optimal performance both the train and test distributions.
Our goal is to learn predictive model
(yl2, (Ior vo)) = Eg (zu 20 ( [Po xz, ye. 2n]]
p(vel2,8e)
that is evact on both the train and test domains when they follow the aforementioned training and testing distribution shifts. We say that learned predictive model j(alb) is E-eracl, eracl on the sample space E, if p(alb) b) V(a b) E E. Here qualify predictive model as robust to test scenario if its ezactness the training domain is sufficient to ensure exactness on the test domain.
We now define an augmented distribution f(ze) over the expert variables whose support ž, includes the joint support Z. Z. between the train and test. distribution of the physical parameters. As depicted Figure& 3, we denote the corresponding support over the observation space x y as for the augmented distribution, Ω for the training distribution, and Ω for the test distribution. In this context, and with A1, we
5

---

Published in Transactions on Machine Learning Research (02/2023)
may demonstrate that even under perfect learning, classical hybrid learning algorithms do not produce Qteeact predictor while our augmentation strategy does.
Assumption 1 (A1): Hybrid modelling learns an interaction model po 2; 2a) that is Q Qeeerac.
Although strong, A1 is consistent with the recent literature on hybrid modelling, which assumes that the expert model (, ze) is an accurate description of the system, thereby the interaction model pa po Mey W, =a) should not be overly complex. As example, consider additive interaction model in our experiments for which extrapolation to unseen Ye holds if the additive assumption is correct. That said, we still notice that the exactness of the interaction model Po on the augmented support Ω is insufficient to prove that the
predictive model Pa.s --eeact Indeed, the encoder dy only trained on the training data and cannot rely on a strong inductive bias in contrast to po. Thus, even the encoder is exact on the training distribution, the corresponding predictive model is not fi-eract. While the decoder's performance not limited to the training scenarios thanks to the broader validity of the expert model, the encoder does not generalize to unseen settings as it is purely data-driven.
3.1 Expert augmentation
We propose data augmentation strategy to improve the robustness of hybrid models to unseen test sce- narios. Once trained, the hybrid model composed of an encoder qs and an interaction model pa that are respectively Ω- and Q-ezact. We may create a new training distribution with support over St by sampling physical parameters 2, from a distribution
A Algorithm 1 Expert augmented hybrid learning
1: D =- (0)) E (X' x )) training set D A parametric encoder An expert model D A parametric interaction model B A hybrid learning objective function A prior distribution on &e that covers both
2: 2e
3: 2())
4: (o(u) Ve, za)
that covers 2. Then, we train the encoder 92 on
5: 11,, U.O,
6: p+ (ze)
the augmented domain Q under perfect training the predictive model (yle, (xo, Mo)) is Ô-ezacl, hence exact on both train and test domains. The expert augmentation is formally described in Algorithm 1, for further details see Appendix A.
train and test scenarios
7: procedure TRAINING
8:
,², arg min E( (1x,, y, 0, +)]
9:
0" is frozen.
10:
D t GENERATEAUUGMENTEDEET
Our learning strategy is grounded in existing hy- brid modelling algorithms, and here, we focus 12:
11:
D R-DUD
Finetuning the encoder on the augmented training set:
APHYNITY and HVAE. We first train an encoder and a decoder Pe with a hybrid learning algo- rithm. Togother with experts we then decide on a realistic distribution p(zz) and create a new dataset ò = ((=e, Fz, ) by sampling from the hybrid generative model defined by Figure 2 and the inter- action model po- A notable difference between the augmented training set D and the original training set D is that the former contains ground truth values for the expert's 's variables ze- As we assume that the interaction model is Ωlezzal, we freeze it and only 22:
13:
8² arg min Ep [[x,, y, 0", ..]]
E+ [log 9n (z.l=,n)) D
14: return 1 8
15:
end procedure
16: procedure GENERATEAUUGMENTEESSE
D ++
18:
for each (Io, E D do
19:
2a ~ qua (Za, Ze Fol Vo)
20:
ze ~ p+ (Ze)
21:
Ve ~ p(ye |x, Ze)
y ~ p (3) x. yes Za)
fine-tune the encoder q6 on DU DUD. We use a combi- 23:
D D { V, 2e)}
nation of the loss function E of the original algorithm
end for
(c.g., Equation (4) for HVAE, and the Lagrangian of Equation (2) for APHYNITY) and supervision the latent variable objective learn decoder
return D
26: end procedure

---

Published in Transactions on Machine Learning Research (02/2023)
that solves
arg min E P
(0(x, y: 8, 0,2)
log go (2
(5)
arg arg min 1
Σ
((2, (1
1
Σ
log qu (zel2.g).
N + N
N
(ny))DDD
(Z..E.y)ED
In our experiments we use a Gaussian distribution for the posterior, which is equivalent to a mean squared error (MSE) loss on the physical parameters. We provide a detailed description of the expert augmentation scheme Appendix A. In principle, the regularizers (the norm of fa for APHYNITY and Rppc, RDA,1 and RD for the Hybrid-VAE) these terms aim to regularize the interaction model. However, in practice, we have observed that this does not matter and makes the implementation even more straightforward.
not necessary Equation (5) only aims to improve the encoder whereas
As a side note, we would like to emphasize the difference between the data augmentation proposed in this paper and the one from Takeishi & Kalousis (2021). While HVAE also requires to sample new physical parameters že, it only to ensure that sub-part of the encoder able to infer correctly ze given ye. This augmentation does not contribute robustness to distribution shifts on y in contrast ours.
4 Experiments
We assess the benefits of expert augmentation on three synthetic problems and one real-world experiment. that are described by the ODE
dyt Fe ze) + Fa(iiiza)
(6)
dt
where F. : 34 x 3, >> 34 is the expert model and Fa 34 2, - 34 complements it. In our notation X is the initial state Yo and the response Y is the sequence of states 91.t1 == / train the models to maximize p (y yıut, x 30) on the training data. We validate and test the models the predictive distribution p(y V1.t2 x yo, To yo, Bo y1.t,), where i2 ti assesses the generalization over time. A brief description of the different problems is provided below.
tt//tt For all experiments we
4.1 Synthetic experiments
The damped pendulum is often used as an example in the hybrid modelling literature (Yin et al., 2021; Takcishi & Kalonsis, 2021). The system's state at time is ye = [ 0. where Og is the angle of the pendulum at time and 0, its angular speed. The evolution of the state over time is described by Equation (6), and
where 2e ==
Fe = [
-wg sin sin 0 and Fg = |
- ab.
(7)
The corresponding systems defined by the damping factor and wo. the fundamental frequency of the pendulum.
The RLC series circuits are electrical circuits made of 3 electrical components that may model a large range of transfer functions. A schomatic such circuit is shown in Figure 10. These models are often used in biology (e.g,, the Hodgkin-Huxe Huxley class models (Hodgkin & Huxley, 1952), in photoplethysmographee (Crabtree Smith, 2003)) and in electrical engineering to model the dynamics of various systems. The system's state time l is ye [U, 1,)", where U, is the voltage over the capacitor and I, the current in the circuit. The evolution of the state over time is described by Equation (6), where Ze == C}, Za = {R} and
4
0
F. =
{(1
U,)
and F.
R
(8)
7

---

Published in Transactions on Machine Learning Research (02/2023)
sin(0,) sin(
11
m
01
0
12
m2
02
0.0
0.5
1.0 Time (second)
1.5
2.0
(a)
(b)
(c)
Figure 4: The double pendulum setup. (a) A photograph of the double pendulum at rest, reproduced from Asseman et al. (2018). (b) simplified sketch setup. (c) An example the time series extracted from videos of the double pendulum.
The dynamics described by the RLC circuit is more diverse than for the pendulum and the system can be hard to identify. This system is characterised by the resistance R, capacitance C, and inductance L, provided Vi is known.
The 2D reaction diffusion was used by Yin al. (2021) to assess the quality APHYNITY.
is a 2D SerHHugh-Nagumnn 32 32 grid. The system's state time is 2 32 32 array yr [ux " The evolution of the state over time is described by Equation (6), where 2e == {a, b), {k} and
Tadu; bAv:
R2 (u,, De; kt; Ru Ut)
Fe
and Fa
(9)
where A is the Laplace operator, the local reaction terms are (u, v; k; = 33 and (u, This model is interesting to study as it considers a state space for which neural architectures may have a real advantage compared to other ML models. In particular, convolutional neural networks are effective for processing signals with spacial and/or temporal correlation.
In these experiments we analyze the effect of our data augmentation strategy on APHYNITY and HVAE. All models explicitly use the assumption that the interaction model follows the structure of Equation (6). For each problem the validation and test sets respectively IID and OOD with respect to the training distribution. The best models always selected based on validation performance, that is with samples from Ω. We provide additional details on the different expert models, dataset creation, and neural networks architectures in Appendix C.
4.2 A real world dataset the double pendulum
We next validate the benefit of the expert augmentation in a controlled real-world setting. The dataset of double pendulum introduced by Asseman al. (2018) contains 21 vidcos of the pendulum shown Figure 4a. Each run lasts approximately 40 seconds and recorded at 400Hz. We can extract the position of the pendulum limbs from each frame with elementary computer-vision tools. Each recording starts from different initial conditions, leading to many states of this chaotic system. For illustration, we showcase the evolution of the arms' angles over time in Figure 4c.
We sketch a simplified representation of the double pendulum in Figure 4b. Its state a four-dimensional vector Di (1) (l) (1) dz())" containing the position and speed of both masses. We can derive
8

---

Published in Transactions on Machine Learning Research (02/2023)
(e.g., (Stachowiak & Okada, 2006)) the kinetics of the frictionless pendulum from first first-principle physics,
0, = -9(2m1 + m22 sin 01 - m2g sin 202) - 2 sin (01 - (2)m2((222 2/22222222 + 01 24 cos(O) - -02)
(10)
(2m + 1/2 m1 cos - -202)
02 2 sin - (0,) 44 (m,, + m2) + gmm, + m2) cos 01 + +02² 1am2 (0) -2))
(11)
12(2m + m2 m2 cos 202))
This ODE is suitable expert model candidate for real-world double pendulum.
We assume that m. = m2- Therefore the effect of masses reduces to constant values in the expert ODE. The length of the two arms are known, 4 = 91mm and l2 = 70mm. The total energy of the double pendulum decreases over time in all videos, which lets us speculate about frictions, not explained by the expert model. In addition, the expert model does not consider potential vibrations or errors in extracting the arms' positions. Hybrid learning has the potential to correct these mispecifications automatically. In comparison, the characterisation of the frictions from first-principle physies is challenging and still research subject (Aghili, 2020).
Similarly to the damped pendulum, we consider the initial angular positions, 0, 0) and ( 0), known. The encoder must predict the initial angular speeds ze := (6 0). (t = 0)} which are the only free parameters of the expert model. The encoder only observes 0, between / = Oms to /. = 100ms and 0₂ between = 50ms to t = 100ms which makes the estimation of ze complicated. Then, we predict the angular positions between 0 and 200ms given (t 0) and 0z(t= = 0) and the cstimation of ze == (01 0), Bz(t 0)} via the hybrid decoder.
We create a dataset with many initial conditions by splitting the videos into consecutive chunks of 20 frames sub-sampled at 100Hz, i.c., 200ms of video. We construct a distribution shift, as shown in Figure 11 from Appendix C.5, over the expert variables ze by splitting each 40 seconds sequence into three parts. The training set only contains chunks from the last 16 seconds of each run. It corresponds to configurations with smaller energy and, thus, slower angular speeds than the test set, which only contains frames from the first 12 seconds. The validation set contains the remaining 12 seconds of frames in the middle.
4.3 Results
Performance gain from augmentation. This experiment demonstrules that HVAE and APHYNITY are not robust to OOD test scenarios in opposition the corresponding AHMs, as shown Figure 1 for the 2D diffusion problem and in Appendix D for the two other problems. We emphasize that our intention is not to declare winner between HVAE and APHYNITY. Indeed, both algorithms have already demonstrated performance superior to black box MI. models. Hence, we only report a very simple baseline that is the mean value of the signals. We want to compare performance in OOD settings and empirically validate the benefit of AHMs. We compare the predictive performance in Figure 5 (see Table 1 for the raw numbers). Although classical hybrid learning strategies do very well on the IID validation set, they exhibit poor generalization on OOD test sets for all three problems. We also observe some disparity between APHYNITY and HVAE. In addition to different. learning strategies, this is probably due to differences in the networks' architectures as they were respectively inspired from the corresponding pendulum experiment in each paper. However, even if one method may outperform the other for some problems, they both benefit from our augmentation strategy (APHYNITY+, HVAE+). Overall, the effect of augmentation goes up to dividing the test error by a factor of c.6 s 100 in some cases.
Stability for non-exact models. The empirical results from Figure 5 are very important as they show that even when the decoder is not P-eract (and hence not Ω-eract), augmentation may still work. In particular, Figure 6 shows that the encoder does not predict the physical parameters perfectly. This indicates that the encoder is not Q-ezact and neither should be the decoder. This plot shows the relative error the = where 14 is the estimated most likely value of the 7th component of the physical parameters. We first notice that APHYNITY and HVAE perform differently and their performance depends on the specific problem. While APHYNITY accurately estimates the physical
physical parameters computed as Σ k
9

---

Published in Transactions on Machine Learning Research (02/2023)
Damped pendulum
RLC series
20 reaction diffusion
Log MAS (lower is better)
AРHYIIT A/HINIY-
Figure The average log-MSEs over runs for three synthetic problems on validation and test sels. We compare HVAE (in red) and APHYNITY (in green), in light colours, to their expert augmented versions HVAE+ and APHYNITY+, in darker colours. On the test sets, A AHMs outperform original models, and by a large margin the pendulum and diffusion problems. Moreover, augmentation conserves the relatively good performance the validation set (TI w.r.t. the training set).
Damped pendulum
RLC series
2D reaction diffusion
APHYNITY
120
APHYNIY+
HARE
Average error
HARE+
Validtton
Figure 6: Comparison of mean relative precision (in %, indicates one standard deviation) over 10 runs of predicted physical parameters of different hybrid modelling strategies in validation and OOD test settings. Augmented versions are denoted with a +. While the accurucy of APHYNITY and HVAE good on the validation set, it collapses on the OOD lest set. On the opposile, the augmented versions perform well both validation and test sels.
parameters on the IID validation set for the 3 problems, HVAF's performance are mixed on the RLC problem as it makes prediction that are around 120% away from the nominal parameter value on average whereas APHYNITY reduces this error to 6%. Interestingly, we observe that the proposed augmentation strategies improve the encoder such that it accurately estimates the physical parameters also on the OOD test set even for HVAE the RLC problem. This confirms that the augmentation strategy is helpful even when the bybrid model is not Q-eract. As a conclusion, augmented hybrid learning outperforms classical hybrid learning both on the predictive accuracy and at inferring the expert variables.
Effect of out of expertise shift. This experiment supports that our augmentation stralegy may remain beneficial even when the train and test supports of za are not identical. This scenario corresponds to samples (x, y) generated by (Ea, =e) E (Z. \ 2a) x 2. depicted by the violet domains in Figure 3. In Figure 7 we observe the log-MSE of augmented and non-augmented hybrid models trained for (#a- ) E Z, x Z, on test data that are generated with (Ea, ze) E 2, x 2.. For the pendulum, the support over z, = a is 0.3] in train and [0.3, 0.6] in test; For the 2D reaction diffusion, =. = is [0.003, 0.005] in train and [0.005, 0.008] in test. We observe that augmented models outperform the original models by a large margin. These results suggest that augmentation valuable if the shift on &, becomes the dominant effect, augmented models also eventually becomes vulnerable to shifts on & as demonstrated by supplementary experiments in Appendix C.
when the distribution shift is not caused by the expert variables. However,
Real-world double pendulum. This experiment demonstrates the potential effectiveness of expert aug- mentation for real-world applications In Figure 9, we compare the empirical performance APHYNITY(+) and HVAE(+) with two baselines: 1) Expert only: the ODE a friction-less double pendulum, 2) ML only: an agnostic neural ODE. We observe Figure that the augmentation improves the validation and test predictive performance by a negligiblee margin, as confirmed visually by Figure 8. In addition, the augmentation improves the estimation of the expert parameters by up to factor of two for APHYNITY in
10

---

Published in Transactions on Machine Learning Research (02/2023)
the OOD test scenarios, shown in Figure 9b. In order to achieve these results, finetune the encoder expert augmentations with 01 ~ 2 [-15, 15] and 02 M (-30, 30].
We have observed that optimizing the HVAE on the double pendulum data eventually becomes numerically unstable along training. It is why we did not manage to obtain HVAE's performance on par with APHYNITY and the MT. only baseline. In consequence, the expert augmentation is also unstable if relies on the best. model. We circumvent these numerical issues by applying the expert augmentation the HVAE model. The expert augmented model HVAE+ outperforms the best HVAE model on predicting the trajectory seen in Figure 9a. Finally, study the effect of applying the expert augmentation at various stages during training for APHYNITY in Appendix D.4. We observe that expert augmentation may reduce the gap between the train train/validation wali/aaiiiaiateeeeeee /test performance on the double pendulum even when the interaction model not fully trained. This result hints again that expert augmentation can help even when the interaction model not learned perfectly.
an earlier version of
Damped pendulum
2D reaction diffusion
1.0
Log-MSE (lower is better)
0.5
-1
sin(B;) sin(0,)
-0.5
APHYNITY APHYNITY+
APHYNITY
APHYNITY
Ground truth
-4
INAE
-1.0
HVNE+
0.00
0.05
0.10 Time (second)
0.15
0.20
-5
Daseline
Figure 8: A cherry-picked example of the predicted angular positions of the double pendulum. We ob- serve that the proposed expert augmentation allows the hybrid model to predict more accurately the state of the double pendulum in the future than the non- augmented hybrid model.
Figure 7: The average log-MSEs over 10 for the damped pendulum and 2D reaction diffussin problems on test distribution for which 2u, addition to žes is also shifted. AHM achieves better peformance than standard algorithmss even when the lest distribution support za differs from the truining.
APHYNITY
HVAE
ML only
175
APHYNITY+
HVAE+
Expert only
150
Relative error
125
Log-MSE
100
75
50
Train
Validation
Test
Train
Validation
Test
(a)
(b)
Figure The results of the double pendulum experiment. (a) The average log-MSE: over three experiments.
The baselines rely either only the expert ODE neural ODE to predict the pendulum's state. The proposed expert augmentation slightly reduces the predictive performance on the training set for APHYNITY but increases the generalisation copobilities of both APHYNITY and the HVAE, APHYNITY+ outperforms the baselines all sets. (b) The average relative errors on the initial angular speeds over three runs. The proposed expert augmentation improves the accuracy of the physical parameters estimation both in the IID and OOD scttings for APHYNITY.
11

---

Published in Transactions on Machine Learning Research (02/2023)
5 Related work
5.1 Hybrid modelling
Hybrid Learning, or gray box modelling as called in its early days in the 90's (Psichogios & Ungar, 1992; Rico-Martinez et al., 1994; Thompson & Kramer, 1994; Rivera-Sampayo & Vélez-Reyes, 2001; Braun & Chaturvedi, 2002), has been a popular method to learn models that are both expressive and interpretable, while also allowing them to be learnt on fewer data. The interest for hybrid learning (Mehta et al., 2021; Lei & Mirams, 2021; Reichstein et al., 2019; Saha et al., 2021; Guen & Thome, 2020; Levine Stuart, 2022; Espcholt et al., 2022) has greatly increased since the outbreak of recent neural network architectures that simplify the combination of physical equations within ML models. As an example, Neural ODEs (Chen et al., 2018) and convolutional neural networks (LeCun et al., 1995, CNN) are privileged architectures to work with dynamical systems described by ODEs or PDEs. While most of the literature focus on the predictive performance of hybrid models, recent work have also shown that this framework helps to infer the physical parameters accurately (Yin et al., 2021; Takeishi & Kalousis, 2021). This is aligned with Zyla et al. (2020) (see Section 40.2.2.2) which observe that inference on incomplete models results in a systematic bias. Similar to hybrid learning, they extend the model with nuisance parameters in order to improve its fidclity, and to reduce the systematic bias.
In this work, decided to study Yin et al. (2021) and Takeishi & Kalousis (2021) for two reasons that distinguish them from the rest of the literature. First, these are notable examples of algorithms that can be applied to a broad class of problems in contrast to papers that focus on specific applications (Lei & Mirams, 2021; Reichstein et al., 2019). Second, those methods also learn a reliable inference model for the physical parameters, suggesting that the expert model is used properly in the generative model, which is a key assumption for our augmentation. While Takeishi & Kalousis (2021) claim to achieve robustness, we argue that this statement is incomplete only able to generalize with respect to unscen time initial state if the model correctly identifies the latent variables Ea; Ze. HVAE camnot generalize to new physical parameters because the encoder's 's validity is bound to the training set for the physical parameters.
HVAE fails in OOD settings. In particular, their approach is
5.2 Combining hybrid modelling and data augmentation
Close to our idea is the one proposed in Shrivastava et al. (2017) where they train a GAN model that improves the realism of a simulated image while conserving its semantic content (e.g,, eyes colour) as modeled by the simulation parameters. The generated data with their annotations may then be used for a downstream task, such as inferring the properties of real images that corresponds to simulation parameters. The GAN objective from Shrivastava et al. (2017) requires that the two distributions induced by the semantic content of real and simulated data are identical. On the opposite, we consider training data that corresponds to export parameters with limited diversity, and overcome this scarcity with expert augmentation. Another line of work similar to ours is Sim2Real, which considers the task of transferring model trained simulated data to real world (Doersch & Zisserman, 2019; Sadeghi et al., 2018a;b). Robust hybrid learning, as way to enhance simulations, could be used for Sim2Real.
5.3 Robust ML and invariant learning
Various statistical methods have been introduced to ensure models generalize under distribution shift. Domain- adversarial objectives aimed at learning (conditionally) invariant predictors (Ganin et al., 2016; Zhang et al., 2017; Li et al., 2018), GroupDRO (Sagawa et al., 2019) optimizing for worst-case loss over multiple domains and IRM (Arjovsky et al., 2019) as well as sub-group calibration (Wald et al., 2021) aiming to satisfy calibration or sufficiency constraints to learn features invariant across domains. Extensions, able to infer domain labels from training data have been proposed as well (Lahoti et al., 2020; Creager et al., 2021), partially inspired by fairness objectives (Hébert-Johnson et al., 2018; Kim et al., 2019). In contrast to AHM, all of these methods rely on the variation of interest being present. in the training data.
12

---

Published in Transactions on Machine Learning Research (02/2023)
6 Discussion
We now discuss the potential limitations of our method and its underlying assumptions.
Erroneous interaction model. The exactness of the hybrid component po Fa) is critical as- sumption underlying our expert-based augmentation strategy. Unfortunately, this component is learned from training data only. Hence, we cannol prove its exactness on the lest domain in the general case as the imput domain over Ve may be different from the training. However, we argue that assumptions on the class of interaction model may alleviate this problem. As an example, we might consider to chose the best interaction model over a fixed set of potential interaction. If the correct interaction is present in this set we should eventually select. it from data. A less extreme example is when we consider an additíve hybrid model and embed this hypothesis into the interaction model, generalization to unseen ye follows as long the range of values is the same as in the training set. If this assumption is too strong, we could still expect that (v xr Ne+ za) generalizes to unseen Ye because hybrid learning drives samples from Py to be close to Ye- It implics that the corresponding function approximator is stable, which helps generalizing to unseen scenarios. However, we must acknowledge that we cannot guarantee the stability of the interaction model in the general case. In practice, finding good inductive bias for the interaction model is important. Moreover, it is usually easier to embed an effective inductive bias in the interaction model rather than in the encoder. Indeed, the parameter identification (the encoder qu) is often less- well understood more complex than the generalive model itself.
Diagnostic. While crucial, we camnot guarantee the exactness of the decoder pe in general because only evaluate the encoder and the decoder jointly on data points (z, Jo, Bo). However, in some cases we can detect model misspecification by observing that the predictive model (o, To. Uo) is imperfect. Making this observation is not always simple as it requires prior knowledge on the expected accuracy of an exact model. However, when the system is identifiable, we may argue that the accuracy should be only limited by the intrinsic measurement noise on y.
Relaxing exactness. Even with solid inductive bias the decoder, achieving exactness is hard in practical settings. However, our experiments demonstrate that expert-angmentation works in practice. We can explain this by looking at Figure 3. If the generative model that maps x and (za, =e) is incorrect, the mapping from Z, and Z, could be slightly off from Ω. However, this does not preclude the set of augmented samples from being closer to Ω than Ω and from inducing a better predictive model on Ω than the original model trained only on ft Another argument is the effectiveness of data augmentation for training classical deep learning models, which works well even when some augmentations do not generate realistic samples. Tn addiiion, Zhang et al. (2022) have observed that the dominant cause of overfitting in classical variational autoencoders is usually coming from the encoder, similarly what happens with amortized hybrid learning. In this regard, an unexpected advantage of expert augmentation would be to reduce this source of overfitting in addition to the OOD robustness gain.
Limitations. We have considered expert models that are parameterized by a small number of parameters and are covered densely via sampling. For higher dimensional parameter space the augmentation strategy might become inapplicable. Hence, a more ingenious sampling strategy, such as worst-case sampling, would be required. Another difficulty is choosing a plausible range parameters that contains both the train and the test support; this will often need a human expert the loop. If the chosen distribution does not cover some test configurations the robustness guarantees for these configurations collapse. On the opposite, a distribution that is too broad may also impact negatively the quality of the model. For instance, learning the right encoding behaviour for those unnecessary configurations consumes representation capacity that is then unavailable for IID configurations. In this case the model might perform worse than before augmentation on the training distributoon. Indeed, some of our results have shown that the training error may slightly increase while the test error decreases.
In addition, we assume that the train distribution of 2 is representative of the test distribution. We empirically observed that a softer version of this assumption could be enough. However, performance will
13

---

Published in Transactions on Machine Learning Research (02/2023)
eventually decline the support of the test distribution for 2a is far from the training domain. Finally, we have only validated our expert augmentation for amortized inference settings. Nevertheless, online inference algorithmss, such as Markov-chain Monte Carlo, also rely on hyperparameters (Campbell et al., 2021) or learnt distributions (Brofos et al., 2022) which must be tuned to the specific problem interest. They could also benefit from expert augmentation if those actionable components are contextually predicted by ML model.
7
Conclusion
We have described hybrid learning within a probabilistic model in which one component of the latent process, denoted the expert model, known. In this context, we have established that state-of-the-art algorithms are vulnerable to distribution shifts. Grounded in this formalisation, we have derived that expert augmentations induce robust.ness to OOD settings. We have shown on a set of synthetic settings and real-world system the favorable effect of expert augmentation on the OOD performance where standard hybrid learning algorithm fails. Finally, we have discussed how our assumptions transfer to real-world settings and have described potential shortcomings.
Our augmentation should benefit from future progress in hybrid learning as shall apply any amortized hybrid modelling algorithms. Providing more substantial constraints on the targeted hybrid model is an essential direction for further improving the robustness of hybrid models. For instance, the minimal description length principle (Grünwald, 2007) could be an excellent resource for investigating the balance between the model's capacity and robustness. In parallel, empirical and theoretical investigations of the learnt interaction model's properties in various practical settings would unlock new understandings of the capabilities and limitations of hybrid learning algorithms. the context of expert augmentation, it should enable to do more realistic assumptions for its applicability and to derive more precisely the expected robustness gain. Future work is also needed lo show how hybrid learning and expert augmentation translate into performance gain over classical ML on challenging real-world applications.
References
Aghili, F. Energetically consistent model of slipping and sticking frictional impacts in multibody systems. Multibody System Dynamics, 48(2):193 209, 2020.
Arjovsky, M., Bottou, T.., Gulrajani, I,, and Lopez-Paz, D. Invariant risk minimization. arXio preprint arXiv:1907.09893, 2019.
Asseman, A., Kornuta, T., and Ozcan, A. Learning beyond simulated physics. In Modeling and mecision-making in the Spatiotemporal Domain Workshop, 2018. URL
Braun, J. E. and Chaturvedi, N. An inverse gray- box model for transient building load prediction. HVACER Research, 8(1):73-9)) 177339992222 2002.
Brofos, J., Gabrié, M., Brubaker, M. A., and Lederman, R. R. Adaptation of the independent metropolis- hastings sampler with normalizing flow proposals. In International Conference on Artificial Intelligence and Statistics, pp. 5949 5986. PMLR, 2022.
Campbell, A., Chen, W., Stimper, V., Hernandez-Lobato, J. M., and Zhang, Y. A gradient based strategy for hamiltonian monte carlo hyperparameter optimization. In International Conference on Machine Learning, pp. 1238 1248. PMLR, 2021.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 32, 2018.
Crabtree, V. P. and Smith, P. R. Physiological models of the human vasculature and photoplethyysmography Electronic Systems and Control Division Research, Department of Electronic and Electrical Engineering, Loughborough Universily, pp. 60 63, 2003.
Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D.,, and Ho, S. Lagrangian neural networks. In
ICLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2020.
14

---

Published in Transactions on Machine Learning Research (02/2023)
Creager, E., Jacobsen, J.-H., and Zemel, R. Environment inference for invariant learning. In International PMLR, 2021.
Conference on Machine Learning, pp.
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Doersch, C. and Zisserman, A. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. Advances in Neural Information Processing Systems, 32: 12:12949-12961, 2019.
Espcholt, L., Agrawal, S., Sønderby, C., Kumar, M., Heck, J., Bromberg, C., Gazen, C., Carver, R., Andrychowicz, M., Hickey, J., Bell, A., and Kalchbrenner, N. Deep learning for twelve hour precipitation forecasts. Nature communications, 13(1):5145, 2022.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):
2096 2030, 2016.
Geirhos, R., Jacobsen, J.-H., Michaclis, C., Zemel, R.,, Brendel, W., Bethge, M., and Wichmann, F. A. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673 2020.
Grünwald, P. D. The Minimum Description Length Principle. MIT press, 2007.
Guen, V. L. and Thome, N. Disentangling physical dynamics from unknown factors for unsupervised vidco prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474-11484, 2020.
Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization, In International Conference Learning Representations, 2020.
Hébert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. Multicalibration: Calibration for the (ommptaatioalll-ddeeiiiabaa masses, In International Conference on Machine Learning, pp. 1939 1948. PMLR, 2018.
Hodgkin, A. L. and Huxley, A. F. A quantitative description of membranc current and its application to conduction and excitation in nerve. The Journal of Physiooogyy 117(4):500 544, 1952.
Keriven, N. and Peyré, G. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32:7092-771 -1011, 2019.
Kim, M. P., Ghorbani, A., and Zou, J. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pp. 247 27-254, 2019.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, L., Gao, I., et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637 5664. PMLR, 2021.
Krucger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. fntt-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F., Thain, N., Wang, X., and Chi, E. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728-740, 2020.
LeCun, Y., Bengio, Y., et al. Convolutional networks for images, speech, and time series. The Handbooo of Brain Theory and Neural Networks, 3361(10), 1995.
Lei, C. L. and Mirams, G. Neural network differential equations for ion channel modelling. Frontiers in Physiology, pp. 1166, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2023)
Levine, M. and Stuart, A. A framework for machine learning of model error in dynamical systems. Commu- nicalions of the American Mathematical Sociely, 2(07):283 2244-344, 2022.
Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., and Tao, D. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 639, 2018.
Mahmood, O., Mansimov, E,, Bonneau, R., and Cho, K. Masked graph modeling for molccule generation. Nature Commmunications, 12(1):1 12, 2021.
Mchta, V., Char, I., Neiswanger, W., Chung, Y., Nelson, A., Boyer, M., Kolemen, E., and Schneider, J. Neural dynamical systems: Balancing structure and flexibility in physical prediction. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 3735 3742. IEEE, 2021.
Psichogios, D. C. and Ungar, L. H. A hybrid neural network-first principles approach to process modeling. AIChE Journal, 38(10):1499 111111 1992.
Qian, Z., Zame, W., Fleuren, L.,, Elbers, P., and van der Schaar, M. Integrating expert odes into neural odes: pharmacology and disease progression. Advances in Neural Information Processing Systems, 34: 11364-11383, 2021.
Reichstein, M., Camps-Vallt G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., al. Deep learning and process understanding for data-driven carth system science. Nature, 566(7743))195-204, 2019.
Rico-Martinez, R., Anderson, J., and Kevrekidis, 1. Continuous-time nonlinear signal processing: a neural network based approach for gray box identification. In Proceedings of IEEE Workshop Neural Networks for Signal Processing, pp. 995- IEFE, 1994.
Rivera-Sampayo, R. and Reyez, M. Gray-box modeling of electric drive systems using neural networks. Tn Proceedings of the 2001 IEEE International Conference on Control Applications (CCA 'Ct) No. 01 01CH37204), pp. 146 151. IEEE, 2001.
Sadeghi, F., Toshev, A., Jang, E., and Levine, Sim2real viewpoint invariant visual servoing by recurrent control. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recogrition, pp. 4691 4699, 2018a.
Sadeghi, F., Toshev, A., Jang, E., and Levine, Sim2real viewpoint invariant visual servoing by recurrent control. In 2018 IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 4691-4699, 2018b. doi:
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2019.
Saha, P., Dash, S., and Mukhopadhyay, S. Plysics-incorporated convolutional recurrent neural networks for source identification and forecasting of dynamical systems. Newral Networks, 144:359 371, 2021.
Shrivastava, A., Pfister, T., Tuzel, O.,, Susskind, J., Wang, W., and Webb, R. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2107-2116, 2017.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. Nature, 450(7676):354-333) 2017.
Stachowiak, T. and Okada, T. A mumerical analysis of chaos in the double pendulum. Chaos, Solitons & Fractals, 29(2):417-425 422, 2006.
Takeishi, N. and Kalousis, A. Pnttere-interratedd variational autoencoders for robust and interpretable generative modeling. Advances in Newral Information Processing Systems, 34, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Thompson, M. L. and Kramer, M. A. Modeling chemical processes using prior knowledge and neural networks. A ATChE Journal, 40(8):1328-1344)) 1994.
Wald, Y., Feder, A., Greenfeld, D., and Shalit, U. On calibration and out-of-domain generalization. Advances in neural information processing systems, 34:2215-225 2227,, 2021.
Willard, J., Jia, X., Xu, S., Steinbach, M., and Kumar, V. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Comput. Sure., 55(4), nov 2022. ISSN 0360-0300. doi: 40.1145/351 /0/3/511222 URL https://doi. heeeiii.ogg/iio wrg/org.///. 11.11/4/355422.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In International Conference on Learning Represenlations, 2019.
Yin, Y., Le Guen, V., Dona, J., de Bézenac, E., Ayed, I., Thome, N., and Gallinari, P. Augmenting physical models with deep networks for complex dynamics forecasting, Journal of Statislical Mechanics: Theory and Experiment, 2021(12):124122 2021.
Zhang, M., Hayes, P., and Barber, D. Generalization gap in amortized inference. In Advances in Newral Information Processing Systems, volume 35, 2022.
Zhang, Y., Barzilay, R., and Jaakkola, T. Transactions of the Association for Compulational Linguistics, 5:515 528, 2017.
adversarial networks for domain adaptation.
Zyla, P. et al. Review of Particle Physics. PTEP, 2080(1) 2020. doi: 10.1093/ 2000///tp/ppaa/
17


Reviews:
[{"review_id": "4YTzXeHB4W", "reviewer": "Reviewer_EcRM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: This submission is very clearly written, and easy to follow even for an audience not familiar with the domain of hybrid learning. The experiments are thorough on multiple domains previously studied in the literature, as well as real-world data. The evaluation metrics are also very clear, and they show an overall boost in performance using the expert augmentation proposed here on both hybrid learning methods (APHYNITY & HVAE).\n\n**Weaknesses**: While the background is very well presented, I think it's unfortunate that the authors don't expand on their contribution as much (Section 3.1), delegating the description of their expert augmentation (the core of their contribution) to the Appendix.\n\nThe method relies on the fact that one knows the true parameters $z_{e}$ of the expert augmented trajectories (to add a likelihood maximization term in the loss), which seems like a strong assumption. In particular, that's probably the reason why the authors had to acquire expert trajectories from a simulated (frictionless) environment for the augmentation on the real-world double pendulum experiment.\n\nI found the use of $\\Omega$-exactness a bit confusing. I understand that it unifies a lot of the concepts (exact over training and/or test settings), but it could probably be presented as one reads instead of having all concepts presented at once in Section 3. For example, in the \"Stability for non-exact models\" (Section 4.3), this could read \"the decoder is not trained perfectly on the training data\" instead of \"the decoder is not $\\Omega$-exact\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the context of hybrid modeling, where an expert mode is augmented by a spearate learned component, the authors propose a novel hybrid data augmentation strategy that improves performance on out-of-distribution scenarios (e.g., physical systems with initial conditions outside of the range encountered during training). The authors show that standard hybrid learning methods fail to learn robustly out-of-distribution. The authors evaluated their expert augmentation strategy on top of 2 different hybrid models (APHYNITY & HVAE), with they thoroughly evaluated it empirically on multiple synthetic domains (damped pendulum, RLC circuit, and 2D reaction diffusion) and data coming from a real-world double pendulum.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggestions:\n - Move the pseudo-code from Appendix A to the main text (in the form of a pseudo-code).\n - Rework the notions of $\\Omega$-exactness (minor suggestion, it can be left as is)", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z7cCeSj7ke", "reviewer": "Reviewer_yBPq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \nThe problem of combing expert knowledge and machine learning is interesting and the presented method shows improvements on several experiments.\n\n# Weaknesses\nThe paper is not well-written and hard to follow, a lot of  necessary information is missing and the methodology introduction is unclear.\n1. The main objective function in Section 3.1  ($l(x,y,\\theta,\\phi)$ + an encoder regularizer) relies on Equations 2 and 4.  Therefore, it is necessary to make a self-contained introduction of both equations and how they relate to the graphical model described in Figure 2. For example, the regularizers in equation 4 should be mathematically described.\n2. In equation 4, there is a regularizer about ``marginal distribution of samples generated by the complete\nmodel to be close to the marginal distribution that would be only generated by the physical model. '', it's not very clear why it makes sense in the OOD setting.\n3. The added regularizer $-\\log q_\\phi(z_e|x,y)$ in section 3.1 is not very clear, the full objective should be presented so we can analyze the consistency.  For example, in the appendix the $z_e$ is sampled from the prior, so the objective should be written as an integration over the $p(z_e)$ etc.. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduce a hybrid data augmentation strategy that can improve the generalization in the OOD setting. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Improve the presentation of the paper on the methodologies, see the weakness part.\n\n1. In Figure 2, the X should be marked (like adding shadow) as an observed variable, otherwise the link between $Y_e\\rightarrow Y$ will be unnecessary.\n\n2. Why assume $p_\\theta(y_e|x,z_e)$ is a delta distribution? It will make the density or the posterior density $p(z_e|x,y_e)$ ill-defined (not absolutely continous). \n\n3. Using the sample pairs $(y_e,z_e)$ from the expert to train the encoder is very similar to the wake sleep training of the amortized inference[1]. Recent work also shows this can improve the in-disrtibution generalization of the encoder[2]. Maybe this connection can help the paper build a more principle explanation of the why the proposed method can improve the generalization in the OOD setting. \n\n\n[1] Dayan P, Hinton G E, Neal R M, et al. The helmholtz machine. Neural computation, 1995\n[2] Zhang M, Hayes P, Barber D. Generalization Gap in Amortized Inference. NeurIPS 2022", "broader_impact_concerns": "I didn't realize any broader impact on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "twskNrWlxb", "reviewer": "Reviewer_Wm5U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* This submission tackles an interesting and important problem. One might mistakenly assume that since hybrid systems incorporate expert knowledge, they are more robust than standard ML algorithms and do not suffer from reduced performance on OOD data. However, this is not the case, and work on addressing robustness in the specific context of hybrid systems is needed.\n\n* This submission proposes a novel solution. Using a probabilistic lens, the authors propose to leverage the expert system further to improve robustness. As far as I can tell, both the author's perspective and proposal contain elements of novelty. \n\n## Weaknesses\n\n(I provide concrete requests to address the weaknesses in the following section.)\n\n* The claims made in the submission are not well supported by evidence. While the submission does include some experimental evidence to support the claims, the experiments were not rigorous enough to give me confidence in the claims. The generality of the work is unclear to me. Furthermore, I have concerns about the reproducibility of the work.\n\n* The submission is not written clearly enough. I struggled to understand some parts of the paper, leaving me with several unresolved questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows that hybrid learning algorithms (those that incorporate a learned parametric model and a domain-specific 'expert' model) can fail when applied to out-of-distribution data. The authors propose to leverage an assumption that expert models are still valid OOD, to provide \"expert augmentation\" which improved OOD performance. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(In addition to [Critical] changes and those that would simply [Strengthen] the submission, I provide a few very [Minor] nitpick requests.)\n\n## Evidence\n\nA. [Critical] Extend the scope of real-world experimental evaluation. Having a single real-world evaluation is not enough to make claims about improved generalization capabilities. I would like to see additional experiments with real-world data. In particular, I would like to see results for cases in which the expert models are more complicated and have a larger number of parameters and cases in which the non-expert model is more complicated (i.e., I would like to see some slightly more modern and large NNs being used to solve complex tasks). It would be great to get some intuition for how the relative complexities of expert and non-expert components impact the performance of expert augmentation. \n\nB. [Critical] Evaluate expert augmentations on HVAE for real-world problems. With only APHYNITY providing evidence in the real-world setting, it is unclear whether expert augmentations apply to general hybrid models in a real-world setting. \n\nC. [Critical] Apply expert augmentations to additional algorithms. In my opinion, even applying expert augmentations to HVAE and APHYNITY is not enough. Alternatively/additionally, provide a more detailed discussion of other hybrid learning algorithms and how exactly they fall within the probabilistic framework presented in Figure 2 (i.e., how expert augmentations can be applied to these other algorithms). Adding as many experimental results as possible would help sell the generality of expert augmentations. \n\nD. [Critical] Provide ablation studies and sensitivity analyses. In particular, I would like to see how the results/conclusions change/remain the same with different model architectures and hyper-parameters. I would also like to see the sensitivity to the sampling ranges/distributions of z_e (e.g., as described in Appendix B). Finally, what happens if the interaction model is not frozen during fine-tuning of the encoder? \n\nE. [Critical] Provide further experimental details. In particular, further discussion about the choices of hyperparameters and the setup for hyperparameter optimisation (where applicable). For example, in appendix B.4, it is stated that an extensive hyperparameter search was required, but no details of that search are given. \n\nF. [Strengthen] Provide a link to the code. In the appendix, there is a reference to more details being in the code, but not code was provided. \n\n## Clarity\n\n1. [Strengthen] I did not understand the sentence \"Because the interaction between z_e and y is essentially defined by the expert model, it should be possible, and preferable, to learn an accurate predictive model of Y whose accuracy is independent from the training distribution of the expert variables z_e\". Please provide some additional explanation/context.  \n\n2. [Strengthen] Provide a derivation for the Bayes optimal predictor in eqn 1.\n\n3. [Strengthen] Please provide some discussion for considering only deterministic expert models. What is the motivation for this choice? What are the consequences? How does this affect generality? (e.g., this means that the KL divergence is not a good choice of distance metric.)\n\n4. [Strengthen] It was slightly confusing that the differential equation for the ODE is written differently in eqns 2 and 3. \n\n5. [Strengthen] Please provide some discussion for only considering distribution shifts in which the marginals for z_a and x are constant. How does this impact the generality of the work? How realistic/common is this setting in practice? \n\n6. [Strengthen] The importance of assumption 1 was unclear to me. I am not sure what the formalism adds to the submission. Furthermore, it seems to me that assumption 1 is actually much stronger than the assumption that \"p(y_e|x, z_e) is an accurate description of the system, thereby p_theta(y|y_e, x, z_a) should not be overly complex\". \n\n7. [Minor]  For the RLC circuit example. \"U_t is the voltage around the capacitance\" should instead be \"U_t is the voltage over the capacitor\". \"V(t)\" is never introduced. There is inconsistency in the notation for indicating time dependency: \"U_t\" and \"I_t\" vs. \"V(t)\". Perhaps a small circuit diagram would be helpful.  \n\n8. [Minor] Using \"array\" rather than \"tensor\" to describe multi-dimensional generalisations of scalars, vectors and matrices is preferred since \"tensor\" has its own meaning in Physics see https://stats.stackexchange.com/questions/198061/why-the-sudden-fascination-with-tensors.\n\n9. [Strengthen] Why does the 2D reaction diffusion considering \"a state space for which neural architectures may have a real advantage compared to other ML models\"? What is the motivation for their potential advantage? \n\n10. [Strengthen] Please motivate why \"soft assumptions on the class of interaction model may alleviate this problem\" (section 6).  ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were all on the fence for this paper.  They agreed that it provided scientifically interesting contributions and was novel.  There were initially concerns regarding clarity, but the authors have already improved these (and have promised to further improve them).  The reviewers did not seem to have any issues with the merits of the technical contribution.  However, there was a discussion about the empirical evidence presented in the paper.  Specifically, the reviewers felt that the experiments were still rather toy (e.g. double pendulum) and the baselines were too weak.  The authors have promised additional experiments i.e. ("We have also added an additional experiment that shows that expert augmentation robustly improves the shifted-test performance.", "We have added results on the double pendulum with HVAE.")

Ultimately, two of three reviewers felt the evidence and claims were justified but only one gave a (weak) accept and two voted for (weak) reject.  One reviewer found the evidence and claims were not justified, given the strength of the experiments.  Given the reviewer recommendations the paper is quite borderline for acceptance.

The majority of reviewers felt the evidence and claims were ok.  Stronger experiments would certainly strengthen the paper and ultimately make it more impactful.  However, the current (and promised) experiments should be sufficient to justify the claims made in the paper.  The reviewers seem to think the paper is correct, technically interesting to some audience of TMLR and innovative.  Following the guidance of TMLR (i.e. "Papers should be accepted if they meet the criteria, even if the contribution or significance of the work is modest.") this paper should be accepted.  I would ask the authors to continue to work on the clarity of the paper (i.e. they said "We have started to address some clarity issues raised by different reviewers" but they should finish doing so.) and incorporate the additional experiments.
Decision: Accept (poster)