Deep Explicit Duration Switching Models for Time Series

1 Introduction . Time series forecasting plays a key role in informing industrial and business decisions [ 17 , 24 , 8 ] , while segmentation is useful for understanding biological and physical systems [ 40 , 45 , 34 ] . State Space Models ( SSMs ) [ 16 ] are a powerful tool for such tasks—especially when combined with neural networks [ 42 , 12 , 13 ] —since they provide a principled framework for time series modeling . One of the most popular SSMs is the Linear Dynamical System ( LDS ) [ 5 , 43 ] , which models the dynamics of the data using a continuous latent variable , called state , that evolves with Markovian linear transitions . The assumptions of LDS allow for exact inference of the states [ 27 ] ; however , they are too restrictive for real-world systems that often exhibit piecewise linear or non-linear hidden dynamics with a finite number of operating modes or regimes . For example , the power consumption of a city may follow different hidden dynamics during weekdays and weekends . Such data are better explained by a Switching Dynamical System ( SDS ) [ 1 , 21 ] , an SSM with an additional set of latent variables called switches that define the operating mode active at the current timestep . Switching events can be classified into time-dependent or state-dependent [ 33 ] . Historically , emphasis was placed on the former , which occurs after a certain amount of time has elapsed in a given regime . While in a vanilla SDS switch durations follow a geometric distribution , more complex long-term ∗Equal contribution . †Work done during an internship at Amazon Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . temporal patterns can be captured using explicit duration models [ 40 , 9 ] . As a recent alternative to time-dependency , recurrent state-to-switch connections [ 35 ] have been proposed that capture state-dependent switching , i.e. , a change that occurs when the state variable enters a region that is governed by a different regime . For added flexibility , these models can be used in conjunction with transition/emission distributions parameterized by neural networks [ 25 , 19 , 13 , 30 ] . Recent works , e.g. , [ 13 , 30 ] , proposed hybrid inference algorithms that exploit the graphical model structure to perform approximate inference for some latent variables and conditionally exact inference for others . Despite these advances in representation and inference , modeling complex real-world temporal phenomena remains challenging . For example , state-of-the-art statedependent models ( e.g. , [ 13 ] ) lack the capacity to adequately capture time-dependent switching . Empirically , we find this hampers their ability to learn parsimonious segmentations when faced with complex patterns and long-term dependencies ( see Fig . 1 for an example ) . Conversely , time-dependent switching models are “ open-loop ” and unable to model state-conditional behavioral transitions that are common in many systems , e.g. , in autonomous or multi-agent systems [ 35 ] . Intuitively , the suitability of the switching model largely depends on the underlying data-generating process ; city power consumption may be better modeled via time-dependent switching , whilst the motion of a ball bouncing between two walls is driven by its state . Indeed , complex real-world processes likely involve both types of switching behavior . Motivated by this gap , we propose the Recurrent Explicit Duration Switching Dynamical System ( RED-SDS ) that captures both state-dependent and time-dependent switching . RED-SDS combines the recurrent state-to-switch connection with explicit duration models for switches . Notably , REDSDS allows the incorporation of inductive biases via the hyperparameters of the duration models to better identify long-term temporal patterns . However , this combination also complicates inference , especially when using neural networks to model the underlying probability distributions . To address this technical challenge , we propose a hybrid algorithm that ( i ) uses an inference network for the continuous latent variables ( states ) and ( ii ) performs efficient exact inference for the discrete latent variables ( switches and counts ) using a forward-backward routine similar to Hidden Semi-Markov Models [ 48 , 9 ] . The model is trained by maximizing a Monte Carlo lower bound of the marginal log-likelihood that can be efficiently computed by the inference routine . We evaluated RED-SDS on two important tasks : segmentation and forecasting . Empirical results on segmentation show that RED-SDS is able to identify both state- and time-dependent switching patterns , considerably outperforming benchmark models . For example , Fig . 1 shows that RED-SDS addresses the oversegmentation that occurs with an existing strong baseline [ 13 ] . For forecasting , we illustrate the competitive performance of RED-SDS with an extensive evaluation against state-ofthe-art models on multiple benchmark datasets . Further , we show how our model is able to simplify the forecasting problem by breaking the time series into different meaningful regimes without any imposed structure . As such , we manage to learn appropriate duration models for each regime and extrapolate the learned patterns into the forecast horizon consistently . In summary , the key contributions of this paper are : • RED-SDS , a novel non-linear state space model which combines the recurrent state-toswitch connection with explicit duration models to flexibly model switch durations ; • an efficient hybrid inference and learning algorithm that combines approximate inference for states with conditionally exact inference for switches and counts ; • a thorough evaluation on a number of benchmark datasets for time series segmentation and forecasting , demonstrating that RED-SDS can learn meaningful duration models , identify both state- and time-dependent switching patterns and extrapolate the learned patterns consistently into the future . 2 Background : switching dynamical systems . Notation . Matrices , vectors and scalars are denoted by uppercase bold , lowercase bold and lowercase normal letters , respectively . We denote the sequence { y1 , . . . , yT } by y1 : T , where yt is the value of y at time t. In our notation , we do not further differentiate between random variables and their realizations . Switching Dynamical Systems ( SDS ) are hybrid SSMs that use discrete “ switching ” states zt to index one of K base dynamical systems with continuous states xt . The joint distribution factorizes as p ( y1 : T , x1 : T , z1 : T ) = T∏ t=1 p ( yt|xt ) p ( xt|xt−1 , zt ) p ( zt|zt−1 ) , ( 1 ) where p ( x1|x0 , z1 ) p ( z1|z0 ) = p ( x1|z1 ) p ( z1 ) is the initial ( continuous and discrete ) state prior . The base dynamical systems have continuous state transition p ( xt|xt−1 , zt ) and continuous or discrete emission p ( yt|xt ) that can both be linear or non-linear . The discrete transition p ( zt|zt−1 ) of vanilla SDS is parametrized by a stochastic transition matrix A ∈ RK×K , where the entry aij = A ( i , j ) represents the probability of switching from state i to state j . This results in an “ open loop ” as the transition only depends on the previous switch which inhibits the model from learning state-dependent switching patterns [ 35 ] . Further , the state duration ( also known as the sojourn time ) follows a geometric distribution [ 9 ] , where the probability of staying in state i for d steps is ρi ( d ) = ( 1− aii ) ad−1ii . This memoryless switching process results in frequent regime switching , limiting the ability to capture consistent long-term time-dependent switching patterns . In the following , we briefly discuss two approaches that have been proposed to improve the state-dependent and time-dependent switching capabilities in SDSs . Recurrent SDS . Recurrent SDSs ( e.g. , [ 6 , 35 , 7 , 30 ] ) address state-dependent switching by changing the switch transition distribution to p ( zt|xt−1 , zt−1 ) —called the state-to-switch recurrence— implying that the switch transition distribution changes at every step and the sojourn time no longer follows a geometric distribution . This extension complicates inference . Furthermore , the first-order Markovian recurrence does not adequately address long-term time-dependent switching . Explicit duration SDS . Explicit duration SDSs are a family of models that introduce additional random variables to explicitly model the switch duration distribution . Explicit duration variables have been applied to both HMMs and SDSs with Gaussian linear continuous states ; the resulting models are referred to as Hidden Semi-Markov Models ( HSMMs ) [ 38 , 48 ] , and Explicit Duration Switching Linear Gaussian SSMs ( ED-SLGSSMs ) [ 9 , 40 , 10 ] , respectively . Several methods have been proposed in the literature for modeling the switch duration , e.g. , using decreasing or increasing count , and duration-indicator variables . In the following , we briefly describe modeling switch duration using increasing count variables and refer the reader to Chiappa [ 9 ] for details . Increasing count random variables ct represent the run-length of the currently active regime and can either increment by 1 or reset to 1 . An increment indicates that the switch variable zt is copied over to the next timestep whereas a reset indicates a regular Markov transition using the transition matrix A . Each of the K switches has a distinct duration distribution ρk , a categorical distribution over { dmin , . . . , dmax } , where dmin and dmax delimit the number of steps before making a Markov transition . Following [ 40 , 9 ] , the probability of a count increment is given by vk ( c ) = 1− ρk ( c ) ∑dmax d=c ρk ( d ) . ( 2 ) The transition of count ct and switch zt variables is defined as p ( ct|zt−1 = k , ct−1 ) = { vk ( ct−1 ) if ct = ct−1 + 1 1− vk ( ct−1 ) if ct = 1 , ( 3 ) p ( zt = j|zt−1 = i , ct ) = { δzt=i if ct > 1 A ( i , j ) if ct = 1 , ( 4 ) where δcond denotes the delta function which takes the value 1 only when cond is true . Although SDSs with explicit switch duration distributions can identify long-term time-dependent switching patterns , the switch transitions are not informed by the state—inhibiting their ability to model state-dependent switching events . Furthermore , to the best of our knowledge , SDSs with explicit duration models have only been studied for Gaussian linear states [ 10 , 9 , 40 ] . 3 Recurrent explicit duration switching dynamical systems . In this section we describe the Recurrent Explicit Duration Switching Dynamical System ( RED-SDS ) that combines both state-to-switch recurrence and explicit duration modeling for switches in a single non-linear model . We begin by formulating the generative model as a recurrent switching dynamical system that explicitly models the switch durations using increasing count variables . We then discuss how to perform efficient inference for different sets of latent variables . Finally , we discuss how to estimate the parameters of RED-SDS using maximum likelihood . 3.1 Model formulation . Consider the graphical model in Fig . 2 ( a ) ; the joint distribution of the counts ct ∈ { 1 , . . . , dmax } , the switches zt ∈ { 1 , . . . , K } , the states xt ∈ Rm , and the observations yt ∈ Rd , conditioned on the control inputs ut ∈ Rc , factorizes as pθ ( y1 : T , x1 : T , z1 : T , c1 : T |u1 : T ) = p ( y1|x1 ) p ( x1|z1 , u1 ) p ( z1|u1 ) · [ T∏ t=2 p ( yt|xt ) p ( xt|xt−1 , zt , ut ) p ( zt|xt−1 , zt−1 , ct , ut ) p ( ct|zt−1 , ct−1 , ut ) ] . ( 5 ) Similar to [ 40 , 9 ] , we consider increasing count variables ct to incorporate explicit switch durations into the model , i.e. , ct can either increment by 1 or reset to 1 at every timestep and represent the run-length of the current regime . A self-transition is allowed after the exhaustion of dmax steps for flexibility . In the subsequent discussion we omit the control inputs ut for clarity of exposition . We model the initial prior distributions in Eq . ( 5 ) for the respective discrete and continuous case as p ( z1 ) = Cat ( z1 ; π ) , ( 6 ) p ( x1|z1 ) = N ( x1 ; µz1 , Σz1 ) , ( 7 ) where Cat denotes a categorical and N a multivariate Gaussian distribution . The transition distributions for the discrete variables ( count and switch ) are given by p ( ct|zt−1 , ct−1 ) = { vzt−1 ( ct−1 ) if ct = ct−1 + 1 1− vzt−1 ( ct−1 ) if ct = 1 , ( 8 ) p ( zt|xt−1 , zt−1 , ct ) = { δzt=zt−1 if ct > 1 Cat ( zt ; Sτ ( fz ( xt−1 , zt−1 ) ) ) if ct = 1 , ( 9 ) where Sτ is the tempered softmax function ( cf . Section 3.3 ) with temperature τ , and fz can be a linear function or a neural network . The probability of a count increment vk for a switch k is defined via the duration model ρk as in Eq . ( 2 ) . The continuous state transition and the emission are given by p ( xt|xt−1 , zt ) = N ( xt ; fµx ( xt−1 , zt ) , fΣx ( xt−1 , zt ) ) , ( 10 ) p ( yt|xt ) = N ( yt ; fµy ( xt ) , fΣy ( xt ) ) , ( 11 ) where fµx , f Σ x , f µ y , f Σ y are again linear functions or neural networks . The model is general and flexible enough to handle both state- and time-dependent switching . The switch transitions zt−1 → zt are conditioned on the previous state xt−1 which ensures that the switching events occur in a “ closed loop ” . The switch duration models ρk provide flexibility to stay long term in the same regime , allowing to better capture time-dependent switching . We use increasing count variables to incorporate switch durations into our model as they are more amenable to the case when the count transitions depend on the control ut . For instance , decreasing count variables , another popular option [ 11 , 36 , 9 ] , deterministically count down from the sampled segment duration length to 1 . This makes it difficult to condition the switch duration model on the control inputs . In contrast , increasing count variables increment or reset probabilistically at every timestep .

Summary: The authors propose Recurrent Explicit Duration Switching Dynamical System model, that is used to detect state and time switching dynamics. The method falls to the category of generalized switching state space model (Ghahramani & Hinton 2000). In particular, they use recurrent state-to-state switching with explicit duration models and provide approximate inference algorithm that is using evidence lower bound. Authors verify they model on 3 segmentation task from time series with ground truth and forecasting task on 5 datasets.