Differentiable Segmentation of Sequences

1 INTRODUCTION . Non-stationarity is a classical challenge in the analysis of sequential data . A common source of non-stationarity is the presence of change points , where the data-generating process switches its dynamics from one regime to another regime . In some applications , the detection of change points is of primary interest , since they may indicate important events in the data ( Page , 1954 ; Box & Tiao , 1965 ; Basseville & Nikiforov , 1986 ; Matteson & James , 2014 ; Li et al. , 2015 ; Arlot et al. , 2019 ; Scharwächter & Müller , 2020 ) . Other applications require models for the dynamics within each segment , which may yield more insights into the phenomenon under study and enable predictions . A plethora of segmented models for regression analysis ( McGee & Carleton , 1970 ; Hawkins , 1976 ; Lerman , 1980 ; Bai & Perron , 2003 ; Muggeo , 2003 ; Acharya et al. , 2016 ) and time series analysis ( Hamilton , 1990 ; Davis et al. , 2006 ; Aue & Horváth , 2013 ; Ding et al. , 2016 ) have been proposed in the literature , where the segmentation materializes either in the data dimensions or the index set . We adhere to the latter approach and consider models of the following form . Let x = ( x1 , ... , xT ) be a sequence of T observations , and let z = ( z1 , ... , zT ) be an additional sequence of covariates used to predict these observations . Observations and covariates may be scalars or vector-valued . We refer to the index t = 1 , ... , T as the time of observation . The data-generating process ( DGP ) of x given z is time-varying and follows a segmented model with K T segments on the time axis . Let τk denote the beginning of segment k. We assume that xt | zt iid∼ fDGP ( zt , θk ) , if τk ≤ t < τk+1 , ( 1 ) ∗corresponding author , e-mail : erik.scharwaechter @ cs.tu-dortmund.de where the DGP in segment k is parametrized by θk . This scenario is typically studied for change point detection ( Truong et al. , 2020 ; van den Burg & Williams , 2020 ) and modeling of non-stationary time series ( Guralnik & Srivastava , 1999 ; Cai , 1994 ; Kohlmorgen & Lemm , 2001 ; Davis et al. , 2006 ; Robinson & Hartemink , 2008 ; Saeedi et al. , 2016 ) , but also captures classification models with concept drift ( Gama et al. , 2013 ) . and segmented generalized linear models ( Muggeo , 2003 ) We express the segmentation of the time axis by a segmentation function ζ : { 1 , ... , T } −→ { 1 , ... , K } that maps each time point t to a segment identifier k. The segmentation function is monotonically increasing with boundary constraints ζ ( 1 ) = 1 and ζ ( T ) = K. We denote all segment-wise parameters by θ = ( θ1 , ... , θK ) . The ultimate goal is to find a segmentation ζ as well as segment-wise parameters θ that minimize a loss function L ( ζ , θ ) , for example , the negative log-likelihood of the observations x . Existing approaches exploit the fact that model estimation within a segment is often straightforward when the segmentation is known . These approaches decouple the search for an optimal segmentation ζ algorithmically from the estimation of the segment-wise parameters θ : min ζ , θ L ( ζ , θ ) = min ζ min θ L ( ζ , θ ) . ( 2 ) Various algorithmic search strategies have been explored for the outer minimization of ζ , including grid search ( Lerman , 1980 ) , dynamic programming ( Hawkins , 1976 ; Bai & Perron , 2003 ) , hierarchical clustering ( McGee & Carleton , 1970 ) and other greedy algorithms ( Acharya et al. , 2016 ) , some of which come with provable optimality guarantees . These algorithms are often tailored to a specific class of models like piecewise linear regression , and do not generalize beyond . Moreover , the use of non-standard optimization techniques in the outer minimization hinders the integration of such models with deep learning architectures , which usually perform joint optimization of all model parameters with gradient descent . In this work , we provide a continuous and differentiable relaxation of the segmented model from Equation 1 that allows joint optimization of all model parameters , including the segmentation function , using state-of-the-art gradient descent algorithms . Our formulation is inspired by the learnable warping functions proposed recently for sequence alignment ( Lohit et al. , 2019 ; Weber et al. , 2019 ) . In a nutshell , we replace the hard segmentation function ζ with a soft warping function γ . An optimal segmentation can be found by optimizing the parameters of the warping function . We propose a novel class of piecewise-constant warping functions based on the two-sided power ( TSP ) distribution ( Van Dorp & Kotz , 2002 ; Kotz & van Dorp , 2004 ) that can represent segmentation functions exactly . TSP-based warping functions are differentiable , have simple closed-form expressions that allow fast evaluation , and their parameters have a one-to-one correspondence with segment boundaries . Source codes for the model and all experiments can be found in the online supplementary material at https : //github.com/diozaka/diffseg . 2 RELAXED SEGMENTED MODELS . We relax the model definition from Equation 1 in the following way . We assume that xt | zt iid∼ fDGP ( zt , θ̂t ) , ( 3 ) where we substitute the actual parameter θk of the DGP at time step t in segment k by the predictor θ̂t . The predictor θ̂t is a weighted sum over the individual segment parameters θ̂t : = ∑ k ŵktθk ( 4 ) with weights ŵkt ∈ [ 0 , 1 ] such that ∑ k ŵkt = 1 for all t. The weights can be viewed as the entries of a ( soft ) alignment matrix that aligns every time step t to a segment identifier k. Given a ( hard ) segmentation function ζ , we can construct a ( hard ) alignment matrix by setting ŵkt = 1 if and only if ζ ( t ) = k. In our relaxed model , we employ a continuous predictor ζ̂t ∈ [ 1 , K ] for the value of the segmentation function ζ ( t ) and let the weight ŵkt depend on the difference between ζ̂t and k : ŵkt : = max ( 0 , 1− ∣∣∣ζ̂t − k∣∣∣ ) ( 5 ) The smaller the difference , the closer the alignment weight ŵkt will be to 1 . If k ≤ ζ̂t ≤ k + 1 , this choice of weights leads to a linear interpolation of the parameters θk and θk+1 in Equation 4 . Higher-order interpolations can be achieved by adapting the weight function accordingly . The key question is how to effectively parametrize the continuous predictors ζ̂t for the segmentation function . For this purpose , we observe that the continuous analogue of a monotonically increasing segmentation function is a warping function ( Ramsay & Li , 1998 ) . Warping functions describe monotonic alignments between closed continuous intervals . Formally , the function γ : [ 0 , 1 ] −→ [ 0 , 1 ] is a warping function if it is monotonically increasing and satisfies the boundary constraints γ ( 0 ) = 0 and γ ( 1 ) = 1 . Any warping function γ can be transformed into a continuous predictor ζ̂t by evaluating γ at T evenly-spaced grid points on the unit interval [ 0 , 1 ] and rescaling the result to the domain [ 1 , K ] . Let ut = ( t− 1 ) / ( T − 1 ) for t = 1 , ... , T be a unit grid . We define ζ̂t : = 1 + γ ( ut ) · ( K − 1 ) . ( 6 ) Similar transformations of warping functions have recently been applied for time series alignment with neural networks ( Weber et al. , 2019 ) . An example segmentation function and predictors based on several warping functions are visualized in Figure 1 . With the transformation from Equation 6 , our relaxed segmented model is parametrized by the segment parameters θ and the parameters of the warping function γ that approximates the segmentation . The discrete-continuous optimization problem from Equation 2 has changed to a continuous optimization problem that is amenable to gradient-based learning : min γ , θ L ( γ , θ ) . ( 7 ) Figure 1 illustrates that the ideal warping function for segmentation is piecewise constant—a formal argument is provided in Appendix A . Several families of warping functions have been proposed in the literature and can be employed within our model ( Aikawa , 1991 ; Ramsay & Li , 1998 ; Gervini & Gasser , 2004 ; Gaffney & Smyth , 2004 ; Claeskens et al. , 2010 ; Freifeld et al. , 2015 ; Detlefsen et al. , 2018 ; Weber et al. , 2019 ; Kurtek et al. , 2011 ; Lohit et al. , 2019 ) . However , none of them contains piecewise-constant functions , and many of them are more expressive than necessary for the segmentation task . Below , we define a novel family of piecewise-constant warping functions that is tailored specifically for the segmentation task , with only one parameter per change point . 3 TSP-BASED WARPING FUNCTIONS . Warping functions are similar to cumulative distribution functions ( cdfs ) for random variables ( Lohit et al. , 2019 ) . Cdfs are monotonically increasing , right-continuous , and normalized over their domain ( Wasserman , 2004 ) . If their support is bounded to [ 0 , 1 ] , they satisfy the same boundary constraints as warping functions . Therefore , we can exploit the vast literature on statistical distributions to define and characterize families of warping functions . Our family of warping functions is based on the two-sided power ( TSP ) distribution ( Van Dorp & Kotz , 2002 ; Kotz & van Dorp , 2004 ) . 3.1 BACKGROUND : TWO-SIDED POWER DISTRIBUTION . The TSP distribution has been proposed recently to model continuous random variables with bounded support [ a , b ] ⊂ R. It generalizes the triangular distribution and can be viewed as a peaked alternative to the beta distribution ( Johnson et al. , 1994 ) . In its most illustrative form , its probability density function ( pdf ) is unimodal with power-law decay on both sides , but it can yield U-shaped and J-shaped pdfs as well , depending on the parametrization . Formally , the pdf is given by fTSP ( u ; a , m , b , n ) = n b−a ( u−a m−a ) n−1 , for a < u ≤ m n b−a ( b−u b−m ) n−1 , for m ≤ u < b 0 , elsewhere , ( 8 ) with a ≤ m ≤ b. a and b define the boundaries of the support , m is the mode ( anti-mode ) of the distribution , and n > 0 is the power parameter that tapers the distribution . The triangular distribution is the special case with n = 2 . In the following , we restrict our attention to the unimodal regime with a < m < b and n > 1 . In this case , the cdf is given by FTSP ( u ; a , m , b , n ) = 0 , for u ≤ a m−a b−a ( u−a m−a ) n , for a ≤ u ≤ m 1− b−mb−a ( b−u b−m ) n , for m ≤ u ≤ b 1 , for b ≤ u . ( 9 ) For convenience , we introduce a three-parameter variant of the TSP distribution with support restricted to subintervals of [ 0 , 1 ] located around the mode . It is fully specified by the mode m ∈ ( 0 , 1 ) , the width w ∈ ( 0 , 1 ] of the subinterval , and the power n > 1 . Depending on the mode and the width , the distribution is symmetric or asymmetric . Illustrations of the pdf and cdf of the three-parameter TSP distribution for various parametrizations can be found in Figure 2 . We denote the three-parameter TSP distribution as TSP ( m , w , n ) and write fTSP ( u ; m , w , n ) and FTSP ( u ; m , w , n ) for its pdf and cdf , respectively . The original parameters a and b are obtained from m and w via a = max ( 0 , min ( 1− w , m− w 2 ) ) , ( 10 ) b = min ( 1 , a+ w ) , ( 11 ) and yield a unimodal regime . Intuitively , the three-parameter TSP distribution describes a symmetric two-sided power kernel of window size w that is located at m and becomes asymmetric only if a symmetric window would exceed the domain [ 0 , 1 ] . An advantage of the TSP distribution over the beta distribution is that its pdf and cdf have closed form expressions that are easy to evaluate computationally . Moreover , they are differentiable almost everywhere with respect to all parameters .

Summary: The paper proposes a relaxed way to solve the segmentation of sequence that can directly leverage the deep learning architectures. The relaxed model allows each segmentation parameter to be a linear interpolations between two consecutive parameters depends on a continuous warping function. The paper then proposes to use mixture of TSP distribution to simulate a step-like warping function and perform a thorough empirical comparison results with different methods and different warping functions. It turns out that TSP-based methods consistently achieve good performance.