Neural Hybrid Automata: Learning Dynamics With Multiple Modes and Stochastic Transitions
Authors: Michael Poli
Stefano Massaroli
Luca Scimeca
Sanghyuk Chun
Seong Joon Oh
Atsushi Yamashita
Hajime Asama
Jinkyoo Park
Animesh Garg
Venue: NeurIPS 2021 Poster (2021)
Keywords: dynamical systems
time series
neural differential equations
control
stochastic hybrid systems

Abstract: Effective control and prediction of dynamical systems require appropriate handling of continuous-time and discrete, event-triggered processes. Stochastic hybrid systems (SHSs), common across engineering domains, provide a formalism for dynamical systems subject to discrete, possibly stochastic, state jumps and multi-modal continuous-time flows. Despite the versatility and importance of SHSs across applications, a general procedure for the explicit learning of both discrete events and multi-mode continuous dynamics remains an open problem. This work introduces Neural Hybrid Automata (NHAs), a recipe for learning SHS dynamics without a priori knowledge on the number, mode parameters, and inter-modal transition dynamics. NHAs provide a systematic inference method based on normalizing flows, neural differential equations, and self-supervision. We showcase NHAs on several tasks, including mode recovery and flow learning in systems with stochastic transitions, and end-to-end learning of hierarchical robot controllers.

Neural Hybrid Automata: Learning Dynamies with Multiple Modes and Stochastic Transitions
Michael Poli" NAVER AI Lab, DiffEqML poli@stanford.edh
Stefano Massaroli" University of Tokyo, DiffEqML
massarolitrobot.co t. u-tookoo.cc...
Luca Scimeca NAVER AI Lab
Scong Joon Oh NAVER AI Lab
Sanghyuk Chun NAVER Al
Atsushi Yamashita University of Tokyo
Hajime Asama University of Tokyo
Jinkyoo Park
Animesh Garg University of Toronto
KAIST
Abstract
Effective control and prediction of dynamical systems require appropriate handling of continuous-time and discrete, event-triggered processes. Stochastic hybrid systems (SHSs), common across engineering domains, provide a formalism for dynamical systems subject discrete, possibly stochastic, state jumps and multi- modal continuous-time flows. Despite the versatility and importance of SHSs across applications, a general procedure for the explicit learning of both discrete events and multi-mode continuous dynamics remains open problem. This work introduces Neural Hybrid Automata (NHAs), recipe for learning SHS dynamics without priori knowledge on the number of modes and inter-modal transition dynamics. NHAs provide a systematic inference method based on normalizing flows, neural differential equations and self-supervision. We showcase NHAs on several tasks, including mode recovery and flow learning in systems with stochastic transitions. and end-to-end learning of hierarchical robot controllers.
Introduction
Behaviors emerging from the interaction of continuous and discrete-time dynamics in the presence of uncertainty are described through the language stochastic hybrid systems (SHSs). Such discrete events bring along abrupt changes in the state, and in complex multi-mode systems, may also cause switch between system modes, and corresponding underlying continuous dynamics [1]. Communication networks [2]. [3], where changes in communication protocol can happen at certain levels of traffic congestion, and biological systems [4]-[6] are example domains where the SHS modeling paradigm has proven fruitful.
Data-driven identification and learning of hybrid systems are known to be challenging due to entanglement of continuous flows and discrete events [7]; finding a generally applicable technique remains an open problem, particularly in the common scenarios where no priori knowledge on the number and type of system modes is given. The aim of this work is to apply continuous neural models [8]-[10] to the learning SHSs. We introduce a compact descriptive language for this task, decomposing the system into set of core primitives. Prior work is integrated into the framework. highlighting in the process limiting assumptions and areas further improvement.
Equual contribution. Author order was decided by flipping coin.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To address the shortcomings of existing techniques, we introduce Neural Hybrid Automata (NHA) as a general procedure designed to enable learning and simulation of SHSs from data. NHAs are comprised of three components: dynamics module, taking the form of an neural differential equation (NDE) [8]-[10] capable approximating different vector field for each mode, a discrete latent state tracking the internal mode of the target system, and an event module determining the time to next event. In particular, our approach does not require prior knowledge on the number of modes. The synergy among NHA components ensures a broader range of applicability compared previous attempts, which in example do not directly tackle multi-mode hybrid systems [1]]-[14] NHAs are shown to enable mode recovery and learning of systems with stochastic transitions, with additional applications in end-to and-to-end learning of hierarchical robot controllers.
2 Background
We introduce required background on the formalism of stochastic hybrid systems (SHSs) and event handling for their numerical simulation. We then provide further contextualization on previous approaches, introducing in the process unified language for SHS learning tasks.
2.1 Stochastic Hybrid Systems
A stochastic hybrid system (SHS) [2]. [15] is right-continuous stochastic process X, taking values in X S - with a latent mode process Ze conditioning the dynamics of Xt, where t 2 0. Z, is another right-continuous stochastic process that takes values in finite set M of size m. In this context, the set M contains identifiers of internal system modes. An event is defined as either a mode switch or a state discontinuity (a jump in Xt), which can in some cases occur simultaneously. We refer to times at which events z + z' occur as random variables I. E T, with associated intensity functions [16]
(1, O.
where H4 :- f4x TT: it <} the history of event times. Intensity, as defined in the classical temporal point process (TPP) sense, can be interpreted as the expected number of events + 2 within the time interval [t. + dt).. The dynamics for X, when Z, defined by
flow dynamics:
d; = F2 (t, xe).
(2,t, xx) E M X
When a jump event 2 + 2' is triggered, X, can instantancously jump according to (z,=',1,4x;)) E M x T x X
jump dynamics:
xxt 21,
Jump maps V/--2' and intensities \د- describe the behavior during events + z'.
2.2 Event Handling for Hybrid Systems
Following [17], to enable forward simulation of SHSs, a convenient mathematical representation of an event is a function g T X E which nullifies only any event time t", thus providing the differential equation integration algorithm with termination or restart condition i.e.
t" is an event < g(t", x(+) O.
(2.1)
The particular form of g induces jump set on DD X determines transitions from roots of to regions the construct enables utilization of root finding methods [18] in neighborhood of t* to accurately zero in on the event time.
g(t, = 0}. and
X where # 0. Notably, this
The same simulation technique can be extended to the many jump sets case typical of multi mode systems, by equipping the condition function with identifiers 2, z' (ga-ze) which induces jump sets Dz-.''
Simulating stochastic events While the event function approach appears to be limited to the deterministic setting, it also subsumes stochastic events whose aleatoric uncertainty is encoded by an intensity X())H)) [1], [13]. Without loss of generality let us consider single intensity function which is henceforth denoted as X; == X(t)H... Recalling that the cumulative distribution function

---

Pay
start
20
Psy
21
Event
20
Psy
21
o' + TextBvent(s)
f111 +- + Va-er
Pesites
(t, 2s). E 04.4-77
Falt, my), E [fu1 +7) Dynamics module
Dynamics module
Event module
Figure 1: Schematic of Neural Hybrid Automata (NHA). The mode- condicoondiioneed Neural ODE F3 drives the system forward until an time f determined by previous call the module. Then, the event module determines flow Pa-+a approximating densities of interevent times. A jump function is then applied to the state ı, and simulation continues with flow Fzr.
to next event and corresponding mode target z' through sampling from normalizing
(CDF) of inter-event times is -ep( Jeo srdt] J. standard inverse transform sampling [1], [19] yields
: 0 5 I didt, Uniform(0,)) 1)
(2.2)
special case of (2.1). Approaches developed for learning TPPs, including in the context of Neural ODEs [8], [11], [20], introduce parametric formulation for the intensity X and optimize via direct TPP likelihood objectives. The integral is in general intractable, thus these methods require either numerical approximation or the augmentation of additional states to compute alongside the ODE.
2.3 Core primitives for SHS learning
At minimum, a learning model for SHSs necessitates several modules, each mirroring an element of the formulation in 2.1. More specifically:
1. Dynamics module, to approximate continuous dynamics F; conditioned on mode 2. d.. Discrete latent selector, to identify at each event time the latent mode of the system.
6d. Event module, to determine when events happen, and how state and latent state are updated after the transition.
Prior work considers specific instantiations of SHSs, leading to simplifying choices for each module defined above. example, switching systems without jumps, where iit. does not require jumps [21], single mode systems, where íi. is not required and tii. does not need determine latent mode transitions [11], [13]. [14], systems with known dynamics, where i. is not trained [13], or systems with only deterministic events [7]. We note some of these works suffer from more than single of these limitations, including additional ones consequence of specific model choices. Direct parametrization of the intensity, while reasonable choice for single mode systems, requires state augmentation scaling in the worst case as (m)) [22] for a SHS with m modes. More importantly, training parameters for such direct approaches affected by the accuracy of the numerical method employed for the solution of the integral in (2.2).
In this work, we introduce modelling framework SHSs that does not rely any of the simplifying assumption on é., ič. and tii. outlined above.
3

---

3 Neural Hybrid Automata
We introduce Neural Hybrid Automata (NHA), model for learning of SHSs. A NHA is comprised of dynamics module, a discrete latent state and an event predictor. A general overview of a NHA is depicted in Figure We start with a description of each module and their interconnections, followed by a step-by stee-by-ssee step procedure for NHA training.
Event module Intensity- free parametrizations for stochastic mode transitions allows NHAs to sample next event times without solving integrals f X; dt for all target modes 2' reachable from current z. From event k at time lx. NHAs determine next event times fx+1 through a conditional normalizing flow modeling, for each possible pair of (z, z'). the density of corresponding inter-event
times Tz = f++1 - tg. t E Tz--''. Let the intensity be a simple timer i.e = 1. Further, let Pz-+=' be the parametrized conditional density obtained by the normalizing flow and let T(2, .ts) be a collection of conditional samples from Pz-+2' (one for each pair 2, z²), i.e.
T(z, (x) = { T2- ~ P2-eer P (T)))s))
2,
Using (2.2), we can thus sample an event given the current mode and the previous event time 4 as:
+ th + min T(2, 24x)
(3.1)
Z'EZ
Note that the next mode 2' after the event is simultaneously obtained as 2' = arg min (22, a', fx) Sampling strategy (3.1), differently from (2.2), relies on the normalizing flow to explicitly model the density rather than defining it implicitly through f Atdr. When event time iy is reached, a parametrized jump map conditioned on (z, z²) is applied to the state x+ = - (f4, 2). Normalizing flow P2-+s' and jump map 32-45' together define the full event module of an NHA.
should be noted that (3.1) always samples the quickest-to-oceut event from the normalizing flow, which implies that other event occurs between f4 and fx+1- While the history can be compressed into a fixed length vector following [23] through application of sequence models e.g. RNNs, we note that for hybrid systems equipped with deterministic events, providing (Ite tx)) as conditioning inputs for Pz->2' (T) is sufficient since ODE solutions with deterministic transitions are uniquely determined by the initial condition. Finally, deterministic events special case of stochastic events [2] that can be well-represented with a Dirac 5 function, of which the normalizing flow learns a smooth approximation with continuous support.
Dynamics module To enable approximation of different mode-deepndentt vector fields, we parametrize the flow map (2tt, xt) of a SHS as a data-controlled neural ordinary differential equation (Neural ODE) [9] with parameters w. driven between each pair of event times (gy lx+1 by discrete latent mode: 2
i. = F1,,
I [lx: (R++1)
(3.2)
Finiteness of admissible values in the latent mode state i.e. ensures F is capable of approximating a finite number of different vector fields, one for each mode. In particular, we consider one-hot representations for latents 2 E RT In batched data settings, (3.2) can be integrated in parallel across n, batches of initial conditions It, E R X with different modes, provided the latent is also batched E 2m
The combination of given dynamics and event module, applied in turn as depicted in Figure 1, enables simulation of trajectories of a SHS. We now describe their training procedure.
4 Neural Hybrid Automata Module Training
Here, we detail the training procedure for each NHA component. Our only assumption availability to trajectory segmentation routine tasked with separating the trajectories, flows, into collection of subtrajectories X, of potentially of different length, each produced by the system in a different mode. The routine can be as simple detection of discontinuities in the solution by inspecting finite- differences of observations across timestamps [24], or involve additional steps such as
detection [25]. Providing exact event times to NHAs is not required; the segmentation routine need only partition the full dataset in n disjoint sets X, s.t. U, X; = X and N, X, = 0. In addition, no knowledge of the number of modes, topology of transitions between modes is made available to NHAs, as these are rarely available in practice.

---

system trajectories
Flow Segmentation
Flow and Mode Seff-Supervisoon
Skmntlly ewezs and segasnl Sows
floe-parallel self supervised mode recovery Via flo ze0oustruction
EA
Train Event and Jump
4
supervised training of paramestiaed
interwes time densitins and jump mops Data labeled following trausisions : + z' deverntined during mode recovery
Simulate sinmlate- prodict
P2-+2
To-tol (x, x
e
Figure 2: NHA training blueprint. Segmenting the trajectories enables self-supervised mode recovery via trajectory reconstructio,, The recovered mode labels are then used for NHA event module supervised training.
Self-supervised mode recovery The first stage of learning an NHA is designed to approximate the continuous dynamics under each SHS mode while simultancously identifying modes We achieve this by framing subtrajectory reconstruction as a pretext task for mode recovery, via a reconstruction objective L, = 1. Ei (X,, X.), being X, subtrajectories reconstructed by the flow decoder Fz via the model
~ E(X,0) i = F1 R2, w)
= tx
(4.1)
LE [|x,4++] (++1).
Here, a latent encoder with parameters 9 is tasked with extracting latent mode state 2 E MI to steer the decoder Fz towards more accurate reconstruction. Representation limitations of Neural ODEs [9], [26] ensure that to fit the above objective the the encoder has to cluster the trajectories to enable the data-control decoder to represent different vector fields for each system mode. Finiteness of admissible values in the latent state enforced by defining z as one-hot encoded sample from a parametrized categorical distribution. Backpropagating through the sampling procedure is performed via
gradients [27]. System (4.1) can be regarded as an ODE trajectory autoencoder with categorical bottleneck.
Finally, we note that trajectory segmentation serves multiple purposes during NHA training. Forward integration is significantly sped up since the ODE solves can now be parallelized across subtrajectories X; as independent samples of a batch of data, avoiding a sequential solve on full SHS traeectories. The speedups can be dramatic for may need to longer to sufficiently explore different modes.
SHSs", the focus of this work, where data trajectories
Event and jump supervision In addition to the learning of mode dynamics, seff-supervieed mode recovery objectives provides direct super- vision for normalizing flows P2-+z' and jump maps s- More specifically, we collect times Tk T and jump state pairs (zz, x+) for each pair of modes (z, z') corresponding to a transition between pairs of subtrajectories clustered as (first) and z' (second) by the encoder E. We then train the jump maps d' to approximate + t,, and the mode conditional normalizing flow to approximate the density (T).
Vai C
Vinl
Vu,C
A = 21,
Figure 3: Conflicting gradients in an idcalized 2 2-mode hybrid system due to overestimation of event time.
Gradient pathologies in joint learning of flows and events When attempting simultaneous learn- ing on the full trajectory, the parameters of flow Fz can be subjected to wrong gradients from reconstruction objectives, arising from overreliance or underreliance of the flow model on certain
"While speedups are dependent on full trajectory and average subtrajectory lengths, in our experiment we observe at least order of magnitude (more than 20x) in wall-clock speedups for single training iteration.
5

---

modes. This phenomenon can bias training, and provides strong motivation behind our segmentation- first approach, since each subtrajectory X; is associated only to single mode".
visualization is provided in Figure 3, through an idealized learning task of a two-mode system. Overreliance of the flow model on mode
0, due to overestimation of event time lg. leads to decomposition of gradients V Wag Lr; in green, gradients pushing the trajectory closer to the solution, in red, incorrect gradients pushing the mode 0 trajectory further away from the ground-truth and closer to solution belonging to a different mode. Appendix A further develops theoretical considerations on the nature of these gradients.
5 Results and Discussion
We evaluate Neural Hybrid Automata (NHA) through extensive experiments, with focus on investi- gating the performance and robustness of each NHA module. summary experiments, objectives and ablations is provided here for clarity:
we carry out a quantitative evaluation on quality of learned flows (mean squared enror) and quality of mode clusters recovered during sulf-super (v-measure). We also verify the robustness NHAs to overclustering and amount of data required for event module training.
Reno TCP:
Mode mixing in switching systems: we highlight and varify robustness against mode mix- ing, a phenomenon occurring during learning of multi-mode systems through alternative soft parametrization of latent 2, such as through sof tmax instead of categorical samples.
Behavioral control of wheeled robots: NHAs enable task- behavioral control. We investi- gate a point-to-poine navigation task where a higher level reinforcement learning (RL) planner determines mode switching for lower-level optimal controller.
5.1 System with Stochastic Transitions
We apply NHAS to a dataset of internal state trajecto- ries of a network transmission controller (TCP), the Reno TCP scheme [2]. The system has two states, five stochastic transitions and three modes as shown through an automata representation in Figure 4. Here, qualitatively validate the performance of dynam- ics and event modules of NHAs. We simulate 40 trajectories of the system, each 200 seconds long, and segment them. No priori knowledge on the mode of each subtrajectory is provided the model. We perform self supervipe mode recovery to train F; and E, in the process labeling the subtrajectories. then train event module normalizing flows and jump functions with the mode labels obtained. Training and evaluation are performed using f-fold cross- validation strategy, with a final test fold of 15. More details on the system, architectures and data genera- tion are reported in the Appendix B.
41 0
start
# 0
(5)
KI1
(1)
#1
(3)
(4)
21
Jumpa (1): [ar 1> [0.693,
(2) [2) + (0.0)
[arg [0.521
(4) (ara + (0.5xe
(6) (21, + (0.0)
Figure 4: Automata representation of TCP Reno,
where 77, Parop, к >> 0 and we set Rack = 2. On each edge is the corresponding intensity \z-22'-
Mode recovery results First, we perform self-supervised mode recovery and verify (i) whether the mode conditioned NHA decoder F. offers test-time TCP trajectory reconstruction of equal or better quality than other Neural ODE variants, and (ii) quality of the mode label clusters assigned by the NHA encoder and robustness to different mumber of latent modes m. We measure (i) via test mean squared error (MSE) on reconstructed trajectories, and (ii) via v-measure [28], a metric taking values in [0, 1]. computed as the harmonic mean between cluster completeness and homogeneity. A v-measure of 1 indicates perfect clustering. As baselines, we collect for (i) the performance of 3 Neural ODE (NODE) variants, zero augmented NODE, data-controlled NODE (DC-NODE: [9] where the latent z is the output of a multi-layer encoder, and a Latent NODE where & is sampled
Due to inaccuracies or noise in the segmentation algorithm, these partitions might not be perfectly separated into different modes. We experimentally investigate these effects NHA training in Appendix B.

---

predicted
Flow and Mode recovery
true
Model
Test RMSE
NODE
3.57
DC-NODE
1.46
30
Latent NODE
1.55
NHA-3
1.66
NHA-5
1.21
NHA-10
0.56
20
40
60
80 100
time (s)
Figure [Left] Reconstruction system trajectories through NHA vector field decoders F3 and corresponding modes z encoded by & for Reno TCP. Although the encoder shown is initialized with more modes (10) than there are in the underlying system (3), mode clustering is sparse and accurate. [Right] Flow reconstruction test RMSE for different classes of decoders, selected via cross-valie vols-validation on 10 runs. NHA decoders can reconstruct the flows as well other NODE baselines, training. NHA-m indicates NHA decoders initialized with m modes.
added benefits being able to recover mode labels during
via reparametrization of a Normal [8]. We also provide baseline performance of series of popular clustering algorithms tasked to cluster the subtrajectories: k-means++ [29]. hierarchical [30] and DBSCAN [31]. Figure 5 provides qualitative and quantitative results for stage (i). As established by Table 1, NHA mode recovery outperforms
baselines by a wide margin, with v-measure values close to 1. Surprisingly, we observe providing the NHA encoder with larger number of latent modes than the 3 of the system improves clustering results and stabilizes training. Additional details on data pre-processing, metrics and baseline design and tuning provided in Appendix B.
Event module results Next, we leverage the mode labels recovered as supervision for the event module of an NHA. In all cases, we train three-layer MLPs as jump maps and two-layer spline flows [32] as nor- malizing flows. Figure 6 visual- izes the learned densities for each stochastic transition for the standard training regime of n = 5 trajec- tories. We also perform an abla- tive study on the quantity of data required extract sufficient super- vision signal for both components of the event module. The results are included in Table 2. We find that a single trajectory is sufficient, with relative performance gains quickly dropping off after n 3.
y-measure 1
Model
m 3
m
5
m
k +
0.20 =
0.24 = 0.02
0.30 0.06
hierarchical
0.23 = 0.01
0.24 - 0.01
0.31 + 0.06
DBSCAN
0.66 = 0.02
0.68 = 0.02
0.69 + 0.01
NHA
0.91 + 0.09
0.95 + 0.04 0.96 + 0.03
Table Quality of recovered made clusters from NHA self- supervised training and baseline clustering algorithms in the TCP task. Hyperparameter m is the number of clusters provided to each algorithu. For DBSCAN, values of m E [3, 5, 10] map instead to its primary parameter E [0.1, 0.5, 1] [31].
Model
Metric
Y =
T1 = 3
n = 5
7
10
P=-+s! We-e!!
NLL MSE (10-3)
2.761
2.375
2.362
2.313
1.435
0.018
0.009
0.003
Table 2: Quality fit for event module components, normalizing flows and jump maps 4b. Training performed with supervising mode labels from n. trajectories TCP. We report test MSE and negative likeiiklddood (NLL) estimated from base normalizing flow model trained ground truth data from n 500.
Interevent
Tamms
densities learned from recovered mode clusters (1, 2)
Ai.
(0,
Si. = (1, 0)
Sc.
(2, 0)
(2,
P. Pe (7)
Deusdy
03-
0.2
112
51
Figure 6: Leafned densities (black) for intervent times T The normalizing flows trained on times recovered during the mode recovery stage, by clustering Try- according the encoder mode labeling. The histogram depicts the ground truth empirical distribution for each class of event.
7

---

5.2 Deterministic Switching System
We investigate mode mixing in three-mode switching linear system (SLS) [13]. The deterministic nature of mode transitions, as well as the absence of state jumps enables direct training of NHAs on data-traecctories without prior segmentation. This allows us to perform an ablative study on mode mixing, phenomenon arising from latent modes produced by the encoder & via soft alternatives to categorical samples, such as by using sof tmax activations.
Mode mixing and overclustering We train NHAs on reconstruction of SL. trajectories. Each encoder is provided, at initialization, one additional latent mode over the three of the system. The conditioned flow F; constructed with three-layer MLPs. Rather than segmenting the data, we repeat sampling for 2 at each integration step. Figure 7 shows the state space switching boundaries and mode vector fields learned by an NHA and a baseline producing z via sof tmax rather than as categorical samples. The additional freedom provided by softmax latents the trajectories by nonlinearly mixing different vector fields to approximate single one. Instead, categorical samples cannot mix the vector fields; this ensures that the learned clustering is either sparse as shown in the Fig.7, 7, or latent values dedicated to the approximation of the same underlying mode dynamics are forced to learn the same vector field. Appendix B contains visualization and analysis for this second case.
R Σ z = 1 allows fitting
In general, categorical bottlenecks are effective when recovery of ground-truth system mode dynamics a primary objective, particularly as it allows pruning of redundant modes as discussed in Appendix B. Sof Smft or other soft relaxations can be viable choice if only black-box fitting of data is desired.
Discrete Mode Selector: Categorical Samples (ftegorical(E.(1n)
2, 3/1
22(2..) y)
kxx,,)
2x(2.))
2
2
35.0 0
0
0
0
-2
27
Discrete Mode Selector: Softmax Weighting (xr
2y (ar, v))
22)
24(2.
2
2
2
-2
-2
-2
2
2
2"
0.9 0
0.5
Figure 7: Reconstructed conditional vector fields F, and corresponding mode classification boundaries in the state-space the LSS. white, the region of state space assigned to mode zg. [Above] Categorical NHA encoder. [Below] Mode classification performed by "soft" encoder & capped with tmax activation. Soft encoders mix vector fields to approximate trajectories, and are thus unable to recover the ground -ruth dynamics for each mode.
5.3 End-To-End Learning of Hierarchical Switching Controllers for Dynamical Systems
Beyond SHS identification, the NHA framework enables learning of task-based hierarchical con- trollers comprising a lowelevel controller u, := u(t, æt, 2) dependent on the discrete mode provided by a higher-level policy æ. Each NHA module adapted as:
dynamics module: d. = F(1, uz)
low-level controlled system
(5.1)
event module: z' < (t, 21,
high-level planner
Within this context, latent state can be regarded as system set-point (e.g. desired value of state .r) determined by the planning policy π to achieve a certain task, which the low-level controller has then to carry out. Both and u, are parametrized by neural networks, and the training is done end-to-end.
8

---

Hierarchical robot control, map
Hierarchical robot control, map
F1
21
Training curves low level and planner controller losses.
80
Loss Eu
60
Rewards
2
40
20
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
Episodes
Figure 8: [Above]] Test-time learned navigation of swarms of differential drive robots. The robots are initialized
random locations and orientations. [Below]: Training loss curves of low-Icvel controller and planner
The obtained hierarchical control scheme is sample efficient, since system dynamics available a priori are included in F. From the perspective я, however, the dynamics are disentangled from the planning objective. Indeed, the higher level policy need only learn how to set and switch between objectives (z + 2'), and not how to control the system to reach them. For this reason we train the model using a loss function partitioned in two terms L = Lu + Lr
Results We consider learning controllers for navigation of two- wheeled differential drive robots [33]. The higher-level model-free policy here trained via REINFORCE gradients [34], [35] to select the nearest resource target every seconds. We set different resources within map, and train all robots to: onc, select the nearest resource target; two, drive the wheels to reach the target. The training of both controllers is carried out concurrently, where target (or mode) selection is performed by π, whereas the behavioural controller u, has to reach the target chosen by æ via low-level steering control inputs. As shown in Figure 8, convergence of both control policies occurs after around 4000 episodes of training. Figure also visualizes the resulting navigation behavior at test-time on two new resource layouts, where alternate between two different sets of targets.
5.4 Generalizable Insights and Empirical Observations
The task of learning SHSs involves several moving components. Ablative experiments have been performed to address specific questions the robustness of NHAs. We detail heuristics that have been observed to improve performance, and report areas of further improvement.
Overclustering stabilizes training We empirically observe providing NHA with more latent modes than the system stabilizes training. We conjecture the additional choice allows the model to use different modes during exploration without always conflicting with other already "assigned" modes, phenomenon which is more frequent, in example, when the number of modes exactly matches that the system. In these cases, dropout in the encoder & appears to improve performance.
Decoder expressivity limitations improve mode recovery Effective mode clustering via the trajectory autoencoder introduced Section 4 relies on representational limitations of Neural ODEs [9]. Ensuring a sufficient state-space density of subtrajectories improves quality of mode clusters.
Noisy segmentation of trajectories We investigate, for the TCP experiment, robustness of mode recovery to incorrect segmentation (Appendix B) and number of NHA latent modes m (Table 1). Extending NHAs to include a finetuning step for trajectory segmentation, in example leveraging ideas from [13] might improve robustness of the segmentation routine and thus overall approach.

---

Related Work
Hybrid system identification and Markov models Hybrid system identification relatively recent development in dynamical system theory [7]. A majority of existing literature focuses on (linear) piecewise affine systems (PWA) [36]. [37]. [38] proposes a clustered symbolic regression algorithm for learning aaput-output maps rather than dynamics. Existing approaches involving continuous optimization [21] do not consider event stochasticity and mode recovery. Identification of SHSs is an even smaller field, with limited success outside specific cases [15].
Continuous-depth and contact models Neural differential equations and continuous- depth mod- els, initially concerned with unimodal systems [8]. [9], have seen preliminary application to the learning of temporal point processes [11]. [39]. Although some of these works tackle stochastic events and marked point processes, multimodality and explicit learning of the flows is not considered. [12] examine interventions as events, and develop a continuous architecture for modeling the lasting effect of given intervention the dynamies. Differentiable contact approaches [14], [40] introduce physics-come compatible models designed to recover deterministic hybrid dynamics of mechanical systems from data. [13] develops, through implicit differentiation, method for direct optimization of event times. Although Neural Event ODEs do not directly address multimodality, a potential synergy between the approach of [13] and NHAs could preserve the advantage of our flow-pand flow-parallel mode recovery, namely integration speed and sidestepping of gradient pathologies outlined Section IV. Section 2.3 provides summary of limitations for these existing methods.
7 Conclusion
Hybrid systems represent versatile and general class of systems, with applications across engineering disciplines [15]. In this work, we investigate challenges related to the learning of SHSs from limited data and introduce Neural Hybrid Automata (NHA), a step-by-ste by-step method leveraging neural differential equations, density estimation and self seff-supee mode recovery. NHAs are shown be effective in various settings, including flow and event learning in systems with stochastic transitions.
Funding Statement
This work was financially supported by NAVER AI Lab, KAIST and University of Tokyo. All experiments were run GPUs provided by KAIST and University of Tokyo.
References
[1] C. G. Cassandras and J. Lygeros, Stochastic hybrid systems. CRC Press, 2018.
[2] J. P. Hespanha, "Stochastic hybrid systems: Application to communication networks," in International Workshop on Hybrid Systems: Computation and Control, Springer, 2004, pp. 387- 401.
[3] G. Sun, C. G. Cassandras, Y. Wardi, C. G. Panayiotou, and G. F. Riley, "Perturbation analysis and optimization of stochastic flow networks," IEEE Transactions on Automatic Control, vol. 49, 12, pp. 2143-2159, 2004.
[4] R. Alur, C. Belta, Ivancic, V. Kumar, M. Mintz, G. Pappas, H. Rubin, J. Schug, and G. Pappas. "Hybrid modeling and simulation of biological systems," Hybrid Systems: Computation and Control, vol. 2034, pp. 19-32, 2001.
[5] J. P. Hespanha and A. Singh, "Stochastic models for chemically reacting systems using polynomial stochastic hybrid systems," International Journal of Robust and Nonlinear Control: IFAC-Affliated Journal, vol. 15, no. 15, pp. 669-689, 2005.
[6] X. Li, O. Omotere, L. Qian, and E. R. Dougherty, "Review of stochastic hybrid systems with applications in biological systems modeling and analysis," EURASIP Journal on Bioinformatics and Systems Biology, vol. 2017, no. 1, pp. 1-12, 2017.
[7] F. Lauer and G. Bloch, "Hybrid system identification," in Hybrid System Identification, Springer, 2019, pp. 77-101.
10

---

[8] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, "Neural ordinary differential equations," in Advances in neural information processing systems, 2018, pp. 6571-6583
[9] S. Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama, "Dissecting neural odes," arXiv preprint arXiv:2002.0800
2020.
[10] C. Rackauckas, Y. Ma, J. Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, and A.
Ramadhan, "Universal differential equations for scientific machine learning," arXiv preprint
arXiv:2001.04333 2020.
[11] J. Jía and A. R. Benson, "Neural jump stochastic differential equations," Advances in Neural Information Processing Systems, 2019, pp. 9843-9854.
[12] D. Gwak, G. Sim, M. Poli, S. Massaroli, Choo, and E. Choi, "Neural ordinary differential equations for intervention modeling." arXiv preprint arXiv:2010.08304, 2020.
[13] R. T. Chen, B. Amos, and M. Nickel, "Learning neural event functions for ordinary differential equations," arXiv preprint arXiv:2011.03900 2020.
[14] Y. D. Zhong, B. Dey, and A. Chakraborty, "A differentiable contact model to extend la- grangian and hamiltonian neural networks for modeling hybrid dynamics," arXiv preprint #rXiv:2122.66944 2021.
[15] C. G. Cassandras and J. Lygeros, "Stochastic hybrid systems," Automation and Control Engineering, vol. 24, 2007.
[16] D. J. Daley and D. Vere-Jones, An introduction to the theory of point processes: volume II: general theory and struciure. Springer Science & Business Media, 2007.
[17] L. Shampine and S. Thompson, "Event location for ordinary differential equations." Computers
& Mathematics with Applications, vol. 39, no. 5-6, pp. 43-54, 2000.
[18] J. Nocedal and S. Wright, Numerical optimization. Springer Science & Business Media, 2006.
[19] J. G. Rasmussen, "Temporal point processes: The conditional intensity function," Lecture Notes, Jan, 2011.
[20] R. T. Chen, B. Amos, and M. Nickel, "Neural spatio-temporal point processes," arXiv preprint arXiv:201 I. I.04583, 2020.
[21] F. Lauer, G. Bloch, and R. Vidal, "A continuous optimization framework for hybrid system identification," Automatica, vol. 47, no. 3, pp. 608-613, 2011.
[22] D. B. West et al., Introduction graph theory. Prentice hall Upper Saddle River, 2001, vol. 2.
[23] O. Shchur, M. Biloš, and S. Günnemann, "Intensity-free learning of temporal point processes," arXiv preprint 1rXiv:1909.12277, 2019.
[24] S. Massaroli, F. Califano, A. Faragasso, M. Risiglione, A. Yamashita, and H. Asama, "Identifi- cation of a class of hybrid dynamical systems," ITAC-PupersOOLiiee, vol. 53, no. 2, pp. 875- 882, 2020.
[25] S. Aminikhanghahi and D. Cook, "A survey of methods for time series change point detection," " Knowledge and information systems, vol. 51, no. 2, pp. 339-367, 2017.
[26] E. Dupont, A. Doucet, and Y. W. Teh, "Augmented neural odes," in Advances in Neural Information Processing Systems, 2019, pp. 3134-3144.
[27] Y. Bengio, N. Léonard, and A. Courville, "Estimating or propagating gradients through stochastic neurons for conditional computation," arXiv preprint arXiv:1308.35 2013.
[28] A. Rosenberg and Hirschberg, "V-measure: A conditional entropy- external cluster evaluation measure," in Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning CEMNLP-CONLL.,
2007, pp. 410-420.
[29] D. Arthur and S. Vassilvitskii, "K-means++: The advantages of careful seeding." Stanford, Tech. Rep., 2006.
[30] F. Murtagh and P. Contreras, "Algorithms for hierarchical clustering: An overview." Wiley Interdisciplinaryy Reviews: Data Mining and Knowledge Discovery, vol. 2, no. 1, pp. 86-97, 2012.
[31] D. Birant and A. Kut, "St-dbscan: An algorithm for clustering knowledge engineering, vol. 60, no. 1, pp. 208-221, 2007.
data," Data &
[32] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, "Neural spline flows," arXiv preprint arXiv: EXX::1990.0403222 2019.

---

[33] S. K. Malu and J. Majumdar, "Kinematics, localization and control of differential drive mobile
robot," Global Journal of Research In Engineering. 2014.
An introduction. MIT press, 2018. [35] J. Peters and S. Schaal, "Policy gradient methods for robotics," in 2006 TEEE/RSJ International
[34] R. S. Sutton and A. G. Barto, Reinforcement learning:
Conference on Intelligent Robots and Systems, IEEE, 2006, pp. 2219-2225.
[36] S. Paoletti, A. L. Juloski, G. Ferrari- Trecate, and Vidal, "Identification of hybrid systems tutorial," European journal of control, vol. 13, no. 2-3, pp. 242-260,, 2007.
[37] R. L. Westra, M. P. Ralf, and L. Pecters, "Identification of piecewise linear models of complex
dynamical systems," IFAC Proceedings Volumes, vol. 44, no. 1, pp. 14 163-1414 868, 2011.
[38] D. L. Ly and H. Lipson, "Learning symbolic representations of hybrid dynamical systems," The Journal of Machine Learning Research, vol. 13, no. I. pp. 3585-3618. 2012.
[39] Y. Rubanova, T. Q. Chen, and D. K. Duvenaud, "Latent ordinary differential equations for inregularly-sampled time series," in Advances in Neural Information Processing Systems, 2019, pp. 5321-5331.
[40] A. Hochlehnert, A. Terenin, S. Sæmundsson, and M. Deisenroth, "Leerning contact dynam- ics using physically structured neural networks," " in International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 2152-2160.
[41] A. L. Mitchell, M. Engelcke, O. P. Jones, D. Surovik, S. Gangapurwala, O. Melon, I. Havoutis, and I. Posner, "First steps: Latent-space control with semantic constraints for quadruped locomotion," " in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2020, pp. 5343-5350.
[42] Y. D. Zhong and N. Leonard, "Unsupervised learning of lagrangian dynamics from images for prediction and control," Advances in Neural Information Processing Systems, vol. 33, 2020.
[43] A. M. Johnson, S. A. Burden, and D. E. Koditschek, "A hybrid systems model for simple manipulation and self- malipmaaiplltioon systems," The International Journal of Robotics Research, vol. 35, no. 11, pp. 1354-1392 1392, 2016.
[44] P. Holmes, R. J. Full, D. Koditschek, and J. Guckenheimer, "The dynamics of legged lo- comotion: Models, analyses, and challenges," SIAM review, vol. 48, no. 2, pp. 207-304, 2006.
[45] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248 -48..55.
[46] E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip, P. Horsfall, and N. D. Goodman, "Pyro: Deep universal probabilistic programming." The Journal of Machine Learning Research, vol. 20, no. I, pp. 973-978, 2019.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., "Scikit-lcarn: Machine learning in python," the Journal of machine Learning research, vol. 12, pp. 2825-2830, 2011.
[48] J. R. Dormand and P. J. Prince, "A family of embedded runge-kutta formulae," " Journal of compulational and applied mathematicy, vol. 6, no. 1, pp. 19-26, 1980.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.4 and Appendix
for discussion on limitations and extensions. The assumptions required are clearly stated in Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes) See the Appendix for details on hardware and compute resources.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "f5l6TwfeTB1", "reviewer": "Reviewer_kV6b", "summary": "This paper introduces a complete pipeline for controlling with multi-modal transition and event handling. The method shown in the paper is based on normalizing flow and neural differential equations. In the experiments, the paper shows the efficacy in a few controlling cases. The method first segments the trajectory into multiple partitions, then performs the reconstruction using normalizing flow.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem formulation and presentation are fine. The figures (especially figure 2) are not really informative enough. The figures and captions should deliver straightforward ideas without ambiguities. The arrows are lack notations or explanations. It will be good to improve them.\n\nThe cross-reference in lines 254 and 259 are broken. The citation format is not uniform. They are not technical problems but it will be helpful to fix them.\n\nMy main reservation is the robustness of the proposed method. Does it perform constantly well against noises? A few quantitative results will help the reader understanding it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Zn9tL53taB", "reviewer": "Reviewer_KEKy", "summary": "This paper considers a modeling class (NHA) based on stochastic hybrid systems that can represent continuous-time processes with discrete events. They discuss the foundations of these models of the dynamics model, discrete latent selector, and event module to select events, and proposed a self-supervised mode recovery and supervision for events and jumps. NHA excel in diverse experimental settings on Reno TCP, mode mixing in switching systems, and behavioral control of wheeled robots. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the paper! The foundations are reasonable and the experiments are well-investigated, analyzed, and documented for the community. \nS5.4 provides a nice overview of their insights and observations, and Fig 7 shows a nice visualization of the learned mode classification boundaries. \n\nMy biggest question about this paper is on the experimental comparisons to neural ODE-based methods --- could NHAs be applied in many of the same settings where NODEs are already being used? The experiments presented in the paper have the feeling of new tasks that haven't been as considered before, which are interesting for the community to think about, but it would also be insightful if these methods could improve upon NODE-based modeling in the existing settings too.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations and insights in S5.4 but otherwise did not have an extensive discussion in this space.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "OxULYYHvufy", "reviewer": "Reviewer_hhU5", "summary": "This paper proposes a solution called 'Neural Hybrid Automata' to model real world dynamical systems with multiple modes of operation and stochastic switching between modes. Such problems are plentiful in a variety of engineering domains. The authors address this using three modules: Dynamics model, Latent model to select the mode and Event model to predict the scitching.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper relies heavily on recent work around the use of Neuro-ODE, Hamiltonian and Lagrangian dynamics. From that perspective this is quite timely and relevant. The maths look good to me. I consider this very interesting, but there are a couple of things that can be improved.\n\n- The first 2 modules of the work are quite solid. However, the last module that focuses on the switching is not as convincing, It is unclear how well it can capture stochasticity for example in a stock price as it reacts to market moving events. This is an area that has been well explored and there exists plentiful literature. Since not enough experiments have been conducted, it is unclear how this extends the state of the are as far as usefulness goes.\n\n- Related to the above, the paper doesn't conduct enough experimentation for real world problems and that in my mind is a limitation. The experiments that are shown with synthetic data are not convincing enough.\n\n- I would also like to see how it compares against known state of the art solutions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations have already been mentioned in the paper. Please see the main review for additional ones.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After clarifications made by the authors in their rebuttal, all reviewers agreed about the merits of this work, and recommended acceptance. In the final version, please incorporate the feedback given in the reviews, and add the important clarifications of the rebuttal.
Decision: Accept (Poster)