Differentiable Segmentation of Sequences

1 INTRODUCTION . Non-stationarity is a classical challenge in the analysis of sequential data . A common source of non-stationarity is the presence of change points , where the data-generating process switches its dynamics from one regime to another regime . In some applications , the detection of change points is of primary interest , since they may indicate important events in the data ( Page , 1954 ; Box & Tiao , 1965 ; Basseville & Nikiforov , 1986 ; Matteson & James , 2014 ; Li et al. , 2015 ; Arlot et al. , 2019 ; Scharwächter & Müller , 2020 ) . Other applications require models for the dynamics within each segment , which may yield more insights into the phenomenon under study and enable predictions . A plethora of segmented models for regression analysis ( McGee & Carleton , 1970 ; Hawkins , 1976 ; Lerman , 1980 ; Bai & Perron , 2003 ; Muggeo , 2003 ; Acharya et al. , 2016 ) and time series analysis ( Hamilton , 1990 ; Davis et al. , 2006 ; Aue & Horváth , 2013 ; Ding et al. , 2016 ) have been proposed in the literature , where the segmentation materializes either in the data dimensions or the index set . We adhere to the latter approach and consider models of the following form . Let x = ( x1 , ... , xT ) be a sequence of T observations , and let z = ( z1 , ... , zT ) be an additional sequence of covariates used to predict these observations . Observations and covariates may be scalars or vector-valued . We refer to the index t = 1 , ... , T as the time of observation . The data-generating process ( DGP ) of x given z is time-varying and follows a segmented model with K T segments on the time axis . Let τk denote the beginning of segment k. We assume that xt | zt iid∼ fDGP ( zt , θk ) , if τk ≤ t < τk+1 , ( 1 ) ∗corresponding author , e-mail : erik.scharwaechter @ cs.tu-dortmund.de where the DGP in segment k is parametrized by θk . This scenario is typically studied for change point detection ( Truong et al. , 2020 ; van den Burg & Williams , 2020 ) and modeling of non-stationary time series ( Guralnik & Srivastava , 1999 ; Cai , 1994 ; Kohlmorgen & Lemm , 2001 ; Davis et al. , 2006 ; Robinson & Hartemink , 2008 ; Saeedi et al. , 2016 ) , but also captures classification models with concept drift ( Gama et al. , 2013 ) . and segmented generalized linear models ( Muggeo , 2003 ) We express the segmentation of the time axis by a segmentation function ζ : { 1 , ... , T } −→ { 1 , ... , K } that maps each time point t to a segment identifier k. The segmentation function is monotonically increasing with boundary constraints ζ ( 1 ) = 1 and ζ ( T ) = K. We denote all segment-wise parameters by θ = ( θ1 , ... , θK ) . The ultimate goal is to find a segmentation ζ as well as segment-wise parameters θ that minimize a loss function L ( ζ , θ ) , for example , the negative log-likelihood of the observations x . Existing approaches exploit the fact that model estimation within a segment is often straightforward when the segmentation is known . These approaches decouple the search for an optimal segmentation ζ algorithmically from the estimation of the segment-wise parameters θ : min ζ , θ L ( ζ , θ ) = min ζ min θ L ( ζ , θ ) . ( 2 ) Various algorithmic search strategies have been explored for the outer minimization of ζ , including grid search ( Lerman , 1980 ) , dynamic programming ( Hawkins , 1976 ; Bai & Perron , 2003 ) , hierarchical clustering ( McGee & Carleton , 1970 ) and other greedy algorithms ( Acharya et al. , 2016 ) , some of which come with provable optimality guarantees . These algorithms are often tailored to a specific class of models like piecewise linear regression , and do not generalize beyond . Moreover , the use of non-standard optimization techniques in the outer minimization hinders the integration of such models with deep learning architectures , which usually perform joint optimization of all model parameters with gradient descent . In this work , we provide a continuous and differentiable relaxation of the segmented model from Equation 1 that allows joint optimization of all model parameters , including the segmentation function , using state-of-the-art gradient descent algorithms . Our formulation is inspired by the learnable warping functions proposed recently for sequence alignment ( Lohit et al. , 2019 ; Weber et al. , 2019 ) . In a nutshell , we replace the hard segmentation function ζ with a soft warping function γ . An optimal segmentation can be found by optimizing the parameters of the warping function . We propose a novel class of piecewise-constant warping functions based on the two-sided power ( TSP ) distribution ( Van Dorp & Kotz , 2002 ; Kotz & van Dorp , 2004 ) that can represent segmentation functions exactly . TSP-based warping functions are differentiable , have simple closed-form expressions that allow fast evaluation , and their parameters have a one-to-one correspondence with segment boundaries . Source codes for the model and all experiments can be found in the online supplementary material at https : //github.com/diozaka/diffseg . 2 RELAXED SEGMENTED MODELS . We relax the model definition from Equation 1 in the following way . We assume that xt | zt iid∼ fDGP ( zt , θ̂t ) , ( 3 ) where we substitute the actual parameter θk of the DGP at time step t in segment k by the predictor θ̂t . The predictor θ̂t is a weighted sum over the individual segment parameters θ̂t : = ∑ k ŵktθk ( 4 ) with weights ŵkt ∈ [ 0 , 1 ] such that ∑ k ŵkt = 1 for all t. The weights can be viewed as the entries of a ( soft ) alignment matrix that aligns every time step t to a segment identifier k. Given a ( hard ) segmentation function ζ , we can construct a ( hard ) alignment matrix by setting ŵkt = 1 if and only if ζ ( t ) = k. In our relaxed model , we employ a continuous predictor ζ̂t ∈ [ 1 , K ] for the value of the segmentation function ζ ( t ) and let the weight ŵkt depend on the difference between ζ̂t and k : ŵkt : = max ( 0 , 1− ∣∣∣ζ̂t − k∣∣∣ ) ( 5 ) The smaller the difference , the closer the alignment weight ŵkt will be to 1 . If k ≤ ζ̂t ≤ k + 1 , this choice of weights leads to a linear interpolation of the parameters θk and θk+1 in Equation 4 . Higher-order interpolations can be achieved by adapting the weight function accordingly . The key question is how to effectively parametrize the continuous predictors ζ̂t for the segmentation function . For this purpose , we observe that the continuous analogue of a monotonically increasing segmentation function is a warping function ( Ramsay & Li , 1998 ) . Warping functions describe monotonic alignments between closed continuous intervals . Formally , the function γ : [ 0 , 1 ] −→ [ 0 , 1 ] is a warping function if it is monotonically increasing and satisfies the boundary constraints γ ( 0 ) = 0 and γ ( 1 ) = 1 . Any warping function γ can be transformed into a continuous predictor ζ̂t by evaluating γ at T evenly-spaced grid points on the unit interval [ 0 , 1 ] and rescaling the result to the domain [ 1 , K ] . Let ut = ( t− 1 ) / ( T − 1 ) for t = 1 , ... , T be a unit grid . We define ζ̂t : = 1 + γ ( ut ) · ( K − 1 ) . ( 6 ) Similar transformations of warping functions have recently been applied for time series alignment with neural networks ( Weber et al. , 2019 ) . An example segmentation function and predictors based on several warping functions are visualized in Figure 1 . With the transformation from Equation 6 , our relaxed segmented model is parametrized by the segment parameters θ and the parameters of the warping function γ that approximates the segmentation . The discrete-continuous optimization problem from Equation 2 has changed to a continuous optimization problem that is amenable to gradient-based learning : min γ , θ L ( γ , θ ) . ( 7 ) Figure 1 illustrates that the ideal warping function for segmentation is piecewise constant—a formal argument is provided in Appendix A . Several families of warping functions have been proposed in the literature and can be employed within our model ( Aikawa , 1991 ; Ramsay & Li , 1998 ; Gervini & Gasser , 2004 ; Gaffney & Smyth , 2004 ; Claeskens et al. , 2010 ; Freifeld et al. , 2015 ; Detlefsen et al. , 2018 ; Weber et al. , 2019 ; Kurtek et al. , 2011 ; Lohit et al. , 2019 ) . However , none of them contains piecewise-constant functions , and many of them are more expressive than necessary for the segmentation task . Below , we define a novel family of piecewise-constant warping functions that is tailored specifically for the segmentation task , with only one parameter per change point . 3 TSP-BASED WARPING FUNCTIONS . Warping functions are similar to cumulative distribution functions ( cdfs ) for random variables ( Lohit et al. , 2019 ) . Cdfs are monotonically increasing , right-continuous , and normalized over their domain ( Wasserman , 2004 ) . If their support is bounded to [ 0 , 1 ] , they satisfy the same boundary constraints as warping functions . Therefore , we can exploit the vast literature on statistical distributions to define and characterize families of warping functions . Our family of warping functions is based on the two-sided power ( TSP ) distribution ( Van Dorp & Kotz , 2002 ; Kotz & van Dorp , 2004 ) . 3.1 BACKGROUND : TWO-SIDED POWER DISTRIBUTION . The TSP distribution has been proposed recently to model continuous random variables with bounded support [ a , b ] ⊂ R. It generalizes the triangular distribution and can be viewed as a peaked alternative to the beta distribution ( Johnson et al. , 1994 ) . In its most illustrative form , its probability density function ( pdf ) is unimodal with power-law decay on both sides , but it can yield U-shaped and J-shaped pdfs as well , depending on the parametrization . Formally , the pdf is given by fTSP ( u ; a , m , b , n ) = n b−a ( u−a m−a ) n−1 , for a < u ≤ m n b−a ( b−u b−m ) n−1 , for m ≤ u < b 0 , elsewhere , ( 8 ) with a ≤ m ≤ b. a and b define the boundaries of the support , m is the mode ( anti-mode ) of the distribution , and n > 0 is the power parameter that tapers the distribution . The triangular distribution is the special case with n = 2 . In the following , we restrict our attention to the unimodal regime with a < m < b and n > 1 . In this case , the cdf is given by FTSP ( u ; a , m , b , n ) = 0 , for u ≤ a m−a b−a ( u−a m−a ) n , for a ≤ u ≤ m 1− b−mb−a ( b−u b−m ) n , for m ≤ u ≤ b 1 , for b ≤ u . ( 9 ) For convenience , we introduce a three-parameter variant of the TSP distribution with support restricted to subintervals of [ 0 , 1 ] located around the mode . It is fully specified by the mode m ∈ ( 0 , 1 ) , the width w ∈ ( 0 , 1 ] of the subinterval , and the power n > 1 . Depending on the mode and the width , the distribution is symmetric or asymmetric . Illustrations of the pdf and cdf of the three-parameter TSP distribution for various parametrizations can be found in Figure 2 . We denote the three-parameter TSP distribution as TSP ( m , w , n ) and write fTSP ( u ; m , w , n ) and FTSP ( u ; m , w , n ) for its pdf and cdf , respectively . The original parameters a and b are obtained from m and w via a = max ( 0 , min ( 1− w , m− w 2 ) ) , ( 10 ) b = min ( 1 , a+ w ) , ( 11 ) and yield a unimodal regime . Intuitively , the three-parameter TSP distribution describes a symmetric two-sided power kernel of window size w that is located at m and becomes asymmetric only if a symmetric window would exceed the domain [ 0 , 1 ] . An advantage of the TSP distribution over the beta distribution is that its pdf and cdf have closed form expressions that are easy to evaluate computationally . Moreover , they are differentiable almost everywhere with respect to all parameters .

Summary: The proposed paper introduces a novel approach for the segmentation of sequences. The proposed method is based on two-sided power distributions (TSP) that are mathematically well-define and enable differentiability. The main goal of the method is to jointly optimize model parameters, including the segmentation function. The method is validated through experiments on modeling the spread of COVID-19 based on Poisson regression, change point detection, a classification model with concept drift (insect stream benchmark), and discrete representation learning (speech signal).