SegTime: Precise Time Series Segmentation without Sliding Window

1 INTRODUCTION . Time series are ordered sequences of data values and they are ubiquitous in a wide range of domains . An important tasks for time series analytics is time series segmentation ( TSS ) ( Wolf et al. , 2006 ; Gensler & Sick , 2014 ) . It refers to the split of data into a number of non-overlapping time segments of possibly different length where each segment corresponds to a given label , that is , data in segments with the same label is similar or follows a similar pattern . Consider an example in Figure 1a where the data are generated as sinusoidal signals and can be split in several segments , where each segment has a frequency and is marked with a predefined colourful label . TSS is essential in various impactful domains , e.g . stocks trajectories partitioning ( Chakraborty et al. , 2016 ) , speaker diarisation ( Wang et al. , 2018 ) , sleep stage labelling ( Perslev et al. , 2019 ) , and human activity recognition ( Chavarriaga et al. , 2013 ; Roggen et al. , 2010 ) . In essence , TSS is a step-level extension of a well known time series classification problem where the latter is regarded by many as one of the most challenging problems in data mining ( Yang & Wu , 2006 ; Esling & Agon , 2012 ) . Indeed , TSS requires to classify segments that are not predefined in advance and there are exponentially many possible segments of a given time series . The common approach to address TSS ( Wang et al. , 2018 ; Perslev et al. , 2019 ; Lee et al. , 2018 ) is to sub-sample the time series using a sliding window of a certain length and overlapping stride , to create sub-time-series of equal length , and then classify these sub-time-series into the given categories . Figure 1b exemplifies this approach by splitting the left part of the input data ( Figure 1a ) into 4 blue overlapping segments ( windows ) that slide with the given stride , and then classified each segment into dark blue , orange and brown categories . After that , these sub-time-series are concatenated together in order to determine the breakpoints between the different categories . The breakpoints are then used to segment the time series according to their categories . In this way , the problem of TSS is reduced to time series classification . This approach is promising when the segment labelling does not change frequently , i.e. , the changing frequency of segment labelling is relatively low compared to sampling rate of measuring sensors as it is in sleep staging where the segment labels are given to every 30s while the sensors sample at 100 Hz in electroencephalography ( Perslev et al. , 2019 ) . However , this approach guarantees to finds only approximate breakpoints since the precise breakpoints can appear in the sub-times-series . The quality of classification with such approach highly depends on the size of the sliding window and the stride , which are difficult to tune ( Yao et al. , 2018 ) . Observe in Figure 1b that the prediction differs a lot from the ground truth ( the breakpoints do not match ) since the windows are large , e.g. , larger than the first segment of the ground truth labelling . Thus , if the segment labelling changes relatively frequently compared to the signal sampling rate , which is quite common in e.g . human activity recognition , industrial machines , driving behaviour detection , then the accuracy will drop drastically . In this work we address the TSS problem and propose a novel neural network approach , SegTime , that can segment time series at time step level and excel both for fast changing labels ( once per tens of samples ) and slow changing labels ( one per thousands of samples ) . This makes SegTime unique , since existing methods are typically tailored towards either fast or slow change but not for both . SegTime has many architectural highlights . We evaluated SegTime extensively against state-of-theart baselines . To summarise , our contributions are as follows : • Conceptual framework . We have several conceptual novelties : ( 1 ) we treat TSS as stepwise classification problem to achieve precise results , while in the literature TSS was commonly treated as time series classification by divide-classify-concatenate ; ( 2 ) we discuss the problem of TSS for two scales of data : fast changing labels ( once per tens of samples ) and slow changing label ( one per thousands of samples ) while existing methods are typically tailored to of them only . • Bi-pass architecture . We propose a bi-pass architecture with following distinguished features : ( 1 ) two core modules : our novel multi-scale skip-LSTM ( long short-term memory ) networks and very deep convolutional neural networks ( CNN ) that both are capable of capturing long-term dependencies ; ( 2 ) several multi-scale pooling ( depthwise separable and atrous pooling ) and skipping structures in the CNN and LSTM that can process information in a multi-scale fashion , that empirically preserves the frequency-invariant representations ; this enables SegTime to cope with time series of a wide spectrum of frequencies ; ( 3 ) a stepwise segmentation module to replace the common practice of sliding window for TSS with step-level prediction of label sequences ( Figure 1c ) . • Evaluation scheme : ( 1 ) we propose a non-trivial transfer of a popular DeepLabv3+ net originally designed for the task of semantic segmentation to a 1D dimension for TSS , which serves as a baseline as well as part of our ablation study1 ; ( 2 ) we demonstrate benefits of our approach with three highly-optimised and sophisticated baselines on two popular datasets . This paper is organised as follows : Sec . 2 reviews related work . Sec . 3 introduces our approach ; Sec . 4 elaborates on the evaluation ; Sec . 5 concludes the paper and previews future directions . 2 RELATED WORK . Impact and History . Time series segmentation is commonly used for a wide range of highly-valued domains ( Bartschat et al. , 2019 ; Waczowicz et al. , 2015 ; ITU , 2012 ) , and has been studied since decades . Early as 2001 , Keogh et al . proposed an online algorithm for TSS applied in aeronautics and cardiography . In 2004 , Keogh et al . surveyed TSS approaches for data of radio waves , exchange rate , manufacturing , civil engineering , space engineering , etc . Such studies are ubiquitous in e.g . finance industry ( Yin et al. , 2011 ; Chakraborty et al. , 2016 ) , music structure analysis ( Serra et al. , 2014 ) , hydrology ( Shao et al. , 2010 ) , ecology ( Meschenmoser et al. , 2020 ; Li et al. , 2021 ) , etc . 1Note this transfer of successful networks from another domain to TSS can bring significant benefit and it is non-trivial . Indeed , observe that a similar transfer of fully convolutional networks to their 1D version ( U-Time ) has been published in Perslev et al . ( 2019 ) and forms the core contribution of that work . TSS Reduced to TSC . Surprisingly , despite of the wide range of applications and long history of study , few recent studies based representation learning ( RL ) directly target on precise TSS , as well as its evaluation on a step level ( Gensler & Sick , 2014 ) . Instead , TSS is often reduced to time series classification ( TSC ) problem ( Wang et al. , 2018 ; Perslev et al. , 2019 ; Lee et al. , 2018 ) . The reason can be that precise TSS is extremely demanding , which in essence requires a step-level time series classification . Considered as one of the most challenging problems in data mining ( Yang & Wu , 2006 ; Esling & Agon , 2012 ) , TSC has received extensive attention in the community . RL-Based TSS on Data with Slow Changing Labels . A series of RL-based works in such dividingclassification-concatenation manner are applied to the problem of sleep staging , which is to segment time series of Electroencephalogram ( EEG ) and electrooculogram ( EOG ) and label each segment with categories of sleep stage names like wake , N1 , N2 , rapid eye movement . The proposed method include CNN-LSTM-based networks ( Supratak et al. , 2017 ) , CNN-based encoder-decoder ( Perslev et al. , 2019 ; 2021 ) , etc . A similar problem is speaker diarisation , where the time series of speech needs to be segmented and labelled with different speaker identities . Many architectures are proposed , such as RNN-based method ( Zhang et al. , 2019 ) , LSTM-based method ( Wang et al. , 2018 ) , LSTM-based encoder-decoder ( Fujita et al. , 2019 ) . These methods may work well on labels of low frequency change . The sampling rate of EEG/EOG signals are 100 Hz while the label is assigned to each 30min of fragment , which means the fastest label change will be slower than once per 3000 time steps . In speaker diarisation , the signal sampling rate is even higher ( 8kHz ) . RL-Based TSS on Data with Fast Changing Labels . Another series of RL-based works are applied to the human activity recognition ( HAR ) , where the label change is of higher frequency . Here the time series need to be split and human activities are assigned to each segment , such as running , walking , standing , sitting , lying , opening or closing the door , turning on or off the light ( Chavarriaga et al. , 2013 ) . The signal sampling rate can be as low as 30Hz , while the label changes can be as fast as once per 20 time steps , since an action of e.g . turning off the light can happen within a short time . Methods on this topic include Classic ML , e.g . k-Nearest Neighbours , Nearest Centroid Classifier ( Gjoreski et al. , 2016 ) , CNN-based ( Yang et al. , 2015 ; Gjoreski et al. , 2016 ) , CNN-LSTMbased ( Wang et al. , 2020 ; Ordóñez & Roggen , 2016 ) , etc . Although the label change frequency is higher , they still divide the time series into fixed-length windows and perform TSC on each window . More Precise TSS . A recent line of work in segmentation aims at avoiding sliding window by segmenting and classifying directly on the input time series via representation learning . Chambers & Yoder ( 2020 ) do so with a fully-connected CNN as the first layer . Another attempt ( Yao et al. , 2018 ) relies on classic ML ( e.g . support vector machines ) and fully-connected CNN , but the segmenting accuracy is sub-optimal and the test set here has labels of lower high frequency changes compared to the training set . Inspiration from Semantic Segmentation . Studies on semantic segmentation shed light on TSS . As a dense structured prediction task , semantic segmentation predicts pixel-level labels for an image ( Chen et al. , 2019 ) . This is very much in line with our goal of step-level labelling . Modules for semantic segmentation like spatial pooling pyramid ( Chen et al. , 2018 ) and ResNet ( He et al. , 2016 ) give us good inspirations . Yet , semantic segmentation is still very different , since some time series are extremely dense ( e.g . 8kHz for speech ) . We still need to rely on time series processing modules , e.g . LSTM ( Gers et al. , 2000 ) . In summary , no prior work has investigated time series segmentation in two scales : labels of high frequency and low frequency change . Few works are dedicated a precise segmentation with steplevel accuracy . These goals are achieved with our solution , SegTime .

Summary: The paper presents a stepwise segmentation for time series data, namely SegTime. Contrary to the sliding window approach, SegTime takes the whole sequence as input, and process it in two separate modules: the MSS-LSTM network and the 1D encoder-decoder network. Outputs from the two separate networks are concatenated and taken as the input to the final convolutional layer to produce the final output, which has class labels for each time-step segment. This paper’s contributions can be summarized as follows:  (1)	By predicting in a step level rather than a sliding window, it works well on both fast- and slow-changing labels.  (2)	High parameter efficiency is achieved with depthwise separable convolution, atrous convolution, and skip-LSTMs. (3)	Long-term dependency is captured, using MSS-LSTM and 1D convolutional layers.