SegTime: Precise Time Series Segmentation without Sliding Window
Authors: Li Zeng
Baifan Zhou
Mohammad Al-Rifai
Evgeny Kharlamov
Venue: ICLR 2022 Submitted (2022)
Keywords: time series
time series segmentation
lstm
rnn
architecture
cnn
pyramid pooling
multi-scale pooling
sequence
encoder
decoder
resnet
step-wise

Abstract: Time series are common in a wide range of domains and tasks such as stock market partitioning, sleep stage labelling, and human activity recognition, where segmentation, i.e. splitting time series into segments that correspond to given categories, is often required. A common approach to segmentation is to sub-sample the time series using a sliding window with a certain length and overlapping stride, to create sub-sequences of fixed length, and then classify these sub-sequences into the given categories. This reduces time series segmentation to classification. However, this approach guarantees to find only approximate breakpoints: the precise breakpoints can appear in sub-sequences, and thus the accuracy of segmentation degrades when labels change fast. Also, it ignores possible long-term dependencies between sub-sequences. We propose a neural networks approach SegTime that finds precise breakpoints, obviates sliding windows, handles long-term dependencies, and it is insensitive to the label changing frequency. SegTime does so, thanks to its bi-pass architecture with several structures that can process information in a multi-scale fashion. We extensively evaluated the effectiveness of SegTime with very promising results.  

Under review as a conference paper at ICLR 2022
SEGTIME: PRECISE TIME SERIES SEGMENTATION WITHOUT SLIDING WINDOW
Anonymous authors
Paper under double-blind review
ABSTRACT
Time series common in wide range of domains and tasks such stock market partitioning, sleep stage labelling, and human activity recognition, where segmen- tation, i.e. splitting time series into segments that correspond to given categories, is often required. A common approach to segmentation is to sub-sample the time series using a sliding window with a certain length and overlapping stride, cre- ate sub-sequences of fixed length, and then classify these sub-sequences into the given categories. This reduces time series segmentation to classification. How- ever, this approach guarantees to find only approximate breakpoints: the precise breakpoints can appear in sub-sequences, and thus the accuracy of segmentation degrades when labels change fast. Also, it ignores possible long-term dependen- cies between sub-sequences. We propose a neural networks approach SegTime Time that finds precise breakpoints, obviates sliding windows, handles long-term de- pendencies, and it is insensitive to the label changing frequency. Seg Time does so, thanks its bi-pass architecture with several structures that can process in- formation in a multi-scale fashion. We extensively evaluated the effectiveness SegTime with very promising results.
INTRODUCTION
Time series are ordered sequences of data values and they are ubiquitous in a wide range of domains. An important tasks for time series analytics is time series segmentation (TSS) (Wolf et al.] (0066] Gensler & Sick 2014). It refers to the split of data into number of non-overlapping time segments of possibly different length where each segment corresponds to given label, that is, data segments with the same label is similar follows similar pattern. Consider an example in Figure [ where the data are generated as sinusoidal signals and can be split in several segments, where each segment has frequency and is marked with predefined colourful label.
TSS is essential in various impactful domains, e.g. stocks trajectories partitioning (Chakraborty et al., 2016), speaker diarisation (Wang et al.] 2018). sleep stage labelling (Perslev al. 2019), and human activity recognition (Chavarriaga et al. [213)] Roggen et al., (0100). In essence, TSS is step-level extension of well known time series classification problem where the latter is regarded
by many as one of the most challenging problems in data mining (Yang & Wu 2006 Esling & Agon 2012). Indeed, TSS requires classify segments that are not predefined in advance and there are cxponentially many possible segments of a given time series.
The common approach to address TSS (Wang et al. 2018 Perslev et al. 2019 Lee al. 2018 is to sub-sample the time series using a sliding window of certain length and overlapping stride, to create sub-time-series of equal length, and then classify these Figure exemplifies this approach by splitting the left part of the input data (Figure TT) into 4 blue overlapping segments (windows) that slide with the given stride, and then classified each segment concatenated
into the given categories.
into dark blue, orange and brown categories. After that, these sub-time-series together in order to determine the breakpoints between the different categorics. The breakpoints are then used to segment the time series according to their categories. In this way, the problem of TSS is reduced to time series classification.
This approach is promising when the segment labelling does not change frequently, i.e., the changing frequency of segment labelling is relatively it is in sleep staging where the segment labels are given every 30s while the sensors sample
low compared to sampling rate of measuring sensors as

---

Under review as a conference paper at ICLR 2022
100 Hz in electroeneephalography PPerseev et al.] 2019). However, this approach guar- antees to finds only approximate breakpoints since the precise breakpoints can appear in the sub-times-serice series. The quality of classification with such approach highly depends on the size of the sliding window and the stride, which are difficult to tune (Yao et al., 2018). Observe in Figure [ that the prediction differs lot from the ground truth (the breakpoints do not match) since the windows are large, e.g., larger than the first segment of the ground truth labelling. Thus, if the segment labelling changes relatively frequently compared to the signal sampling rate, which is quite common in e.g. human activity recognition, industrial machines, driving behaviour detection, then the accuracy will drop drastically.
breakpoints
input
time
M/
series
label
(ground
-truth)
alkding window
stapwise sogmen- talion
wi wrndoww
weigh O
classif-
cation
predicted
labels
Figure 1: (a) Time series segmentation; (b) sliding
window approach; (c) our approach
In this work we address the TSS problem and propose a novel neural network approach, SegTime, that can segment time series time step level and excel both for fast changing labels (once per tens of samples) and slow changing labels (one per thousands of samples). This makes SegTime unique, since existing methods are typically tailored towards either fast or slow change but not for both. SegTime has many architectural highlights. We evaluated Seg Time extensively against state-of-the- art baselines. To summarise, our contributions are as follows:
Conceptual framework. We have several conceptual novelties: (1) we treat TSS as stepwise clas- sification problem achieve precise results, while in the literature TSS was commonly treated as time series classification by livide-clasiff--cocaaeeme two scales of data: fast changing labels (once per tens of samples) and slow changing label (one per thousands of samples) while existing methods are typically tailored to of them only.
(2) we discuss the problem of TSS for
Bi-pass architecture. We propose bi-pass architecture with following distinguished features: (1) two core modules: our novel multi-scale skip-LSTM (long short-term memory) networks and very deep convolutional neural networks (CNN) that both are capable of capturing long- term de- pendencies; (2) several multi-scale pooling (depthwise separable and atrous pooling) and skipping structures the CNN and LSTM that can process information in a multi-scale fashion, that empir- ically preserves the frequency- invariant representations; this enables SegTime to cope with time series of wide spectrum of frequencies; (3) a stepwise segmentation module to replace the com- mon practice of sliding window for TSS with step-level prediction of label sequences (Figure [b). Evaluation scheme: (1) we propose non trivial transfer a popular DeepLabv3+ net originally designed for the task of semantic segmentation to a ID dimension for TSS, which serves baseline as well as part of our ablation study (2) we demonstrate benefits of our approach with three highly-optimised and sophisticated baselines two popular datasets.
This paper organised as follows: Sec. 2 reviews related work. Sec. 3 introduces our approach; Sec. 4 elaboortee on the evaluation; Sec. concludes the paper and previews future directions.
RELATED WORK
Impact and History. Time series segmentation is commonly used for a wide range of highly-
domains (Bartschat et al. 2019 Waczowicz al.) 2013) ITU 2012), and has been studied since decades. Early as 2001, Keogh et al. proposed an online algorithm for TSS applied in aeronautics and cardiography. 2004, Keogh et al. surveyed TSS approaches for data of radio waves, exchange rate, manufacturing. civil engineering, space engineering, etc. Such studies are ubiquitous in e.g. finance industry (Yin et al., 2011; Chakraborty et 2016)) music structure analysis (Serra et al. 2014), hydrology (Shao et al.) 2010). ecology (Meschenmoser et al. 2020 Li et al. 2021). etc.
Note this transfer of successful networks from another domain to TSS can bring significant benefit and it is non-trivial. Indeed, observe that similar transfer of fully convolutional networks to their 1D version (U-Time) has heen published in Perslev al. (2019) and forms the core contribution of that work.
2

---

Under review as a conference paper at ICLR 2022
TSS Reduced to TSC. Surprisingly, despite of the wide range of applications and long history of study, few recent studies based representation learning (RL) directly target on precise TSS, as well as its evaluation
step level (Gensler & Sick 2014). Instead. TSS is often reduced to time series classification (TSC) problem (Wang et al, 2018 Perslev et al. 2019 Lee et al. 2018). The reason can be that precise TSS is extremely demanding. which in essence requires step-level time series classification. Considered as one of the most challenging problems in data mining (Yang & Wu. 2006 Esling & Agon 2012), TSC has received extensive attention in the community.
RL-Based TSS on Data with Slow Changing Labels. A series of RL-based works in such dividing- olassfication-concateaaaao manner are applied the problem of sleep staging, which is to segment time series of Electroenceehalograa
(EEG) and electrooculogram (EOG) and label each segment with categories of sleep stage names like wake, N1, N2, rapid eye movement. The proposed method include CNN-LSTM-based networks (Supratak et al. 2017), CNN-based encoder-decoder (Perslev) al. 2019 202T), etc. A similar problem is speaker diarisation, where the time series of speech needs to be segmented and labelled with different speaker identities. Many architectures are pro- posed, such as RNN- based method (Zhang et all 2019). LSTM-based method (Wang et al. 2018)) LSTM-based encoder-decodee frequency change. The sampling rate of EEG/EOG signals are 100 Hz while the label is assigned each 30min of fragment, which means the fastest label change will be slower than once per 3000 time steps. In speaker diarisation, the signal sampling rate is even higher (8KHz).
(Fujita et al. 2019). These methods may work well labels of low
RL-Baased TSS Data with Fast Changing Labels. Another series of RL-based works ap- plied to the human activity recognition (HAR), where the label change is of higher frequency. Here the time series need to be split and human activities are assigned to each segment, such as running, walking, standing, sitting, lying, opening or closing the door, turning on or off the light (Chavarriagal al. 2013). The signal sampling rate can be as low as 30Hz, while the label changes can be as fast as once per 20 time steps, since an action of e.g. turning off the light can happen within a short time. Methods on this topic include Classic MI., e.g. k-Nearest Neighbours, Nearest Centroid Classifier (Gjoreski et al. 2016), CNN-based (Yang et al. 2013 Gioreski et al., 2016), CNN-LSTM- based (Wang et al. 2020; Ordóñez& & Roggen 2016), etc. Although the label change frequency is higher, they still divide the time series into fixed-length windows and perform TSC each window.
More Precise TSS. A recent line of work segmentation aims at avoiding sliding window by segmenting and classifying directly on the input time series via representation learning. Chambers & Yoder (220)) do so with a fully-comnected CNN as the first layer. Another attempt (Yao et al., 2018) relies classic MI. (e.g. support vector machines) and fully-connected CNN, but the segmenting accuracy is sub-optimal and the test set here has labels of lower high frequency changes compared to the training set.
Inspiration from Semantic Segmentation. Studies on semantic segmentation shed light on TSS. As dense structured prediction task, semantic segmentation predicts pixel-level labels for an im- age (Chen et al. 2019). This is very much in line with our goal of step-level labelling. Modules for semantic segmentation like spatial pooling pyramid (Chen al. 2018) and ResNet (He et al. 2016) give us good inspirations. Yet, semantic segmentation is still very different, since some time series are extremely dense (e.g. 8kHz for speech). We still need to rely on time series processing modules, e.g. LSTM (Gers et al. 2000).
In summary, no prior work has investigated time series segmentation in two scales: labels of high frequency and low frequency change. Few works are dedicated precise segmentation with step- level accuracy. These goals are achieved with our solution, SegTimo
3 OUR APPROACH: SEGTIME
We now formulate the problem of time series segmentation, and introduce our approach Segeime.
Definitions and Problem Statement. A (univariate) time series X = [xi, 21.] is an ordered sequence of real numbers, where x, is the value at the time step t, and L is the total length. We use the term sequence interchangeably with time series when it is more convenient. X can have a stepwise corresponding label sequence y = [y1, V2. ...: yL], which assigns a sequence of label values to X; here Y/ is the label corresponding to Zt. Continuing with the example in Figure I a), it can be represented with a time series X of length, say 10", where the first 100 elements 2;,8 correspond to the fragment of the sinusoidal signal which is marked with blue and
3

---

Under review as a conference paper at ICLR 2022
input W
SKip-LSTM
K=1
1D-DS ResNet
SKIp-LSTM
Encoder
K=2
output sequence
MSS- LSTM Net
Upsampling
AMSP
SKip-LSTM
Dropout
K=4
Upeampling
SKip-LSTM
Decoder
K-8
Upsampling
Concatenation
Input
sequence
Stepwise Segmentation
MSS-LSTM
label
Net
Figure 2: (a) Architecture of SegTime (An expanded overview is given in Appendix Figure (b) Multi-Scale Skip LSTM Net (MSS-LSTM) Blue rounded rectangles indicate complex neutral networks, and green rounded rectangles indicate simple layers. In the skip-LSTMs, k is the skipping factor. When k = 1, a SKip-LSTTM equal a normal LSTM.
thus corresponding y:s from 1 to 100 are equal to "blue". A multivariate time series of size M. X X2,, Xz], consists of M univariate time series. X can have stepwise corresponding label sequence y. where X, = 2f, MM is the vector values x all time series at time step 1, which corresponds to the label y, time step
The goal of time series segmentation is to perform stepwise classification of X (or X XM using a predefined set of labels yys that yields the label sequence y. This in particular allows precisely determine breakpoints, that is, the values of j such that # Dj+1-
Architectural Overview. SegTime consists of five modules and they are organised in a bi-pass architecture (Figure ()). The input sequence go through two passes: an MSS-LSTM Net (Multi-scale skip LSTM) and 1D-encoder-
input sequence
Decoder
Encoder
ID-DS-ResNet
1D Conv
Low-lavel
decoder module. The latter one consists of three sub-modules wise separable), AMSP (atrous multi-scale pooling) (these two form the encoder), and the decoder. The outputs of the encoder-decoder module and MSS-LSTM Net are then concate- nated. Then they go through mentation module, which can predict the output labels in the step-level, thus achieving precise time series segmentation.
ID-DS-ResNet (depth-
Upsampling
Concatenation
stepwise Seg-
1D-Conv
Concatenation
MSS-LSTM Net. We propose the MSS-LSTM Net (Multi-Scale Skip LSTM, Figure 2b). In MSS-LSTM. the input sequence goes through multiple LSTM units, which have different scales of skipping (inspired by Lai et (2018). We use the hyper-parameter le to in- dicate the skipping factor (it can also under- stood down-sampling rate), namely by every &: value the Skip-LSTM takes value. The out- puts of the multiple sKip-LSTM are upsampled and concatenated, and then go through an optional dropout layer. Note that the numbers of skip-LSTMs of different skipping factors independent syper-parametert They can be tuned to target on data different scales. For example, if the la- bel changes are of high frequency, i.e. when the input sequence sampled with relatively low
1D-Conv
Upsampling
output sequence
Figure 3: The -D-encoder-decoder structure

---

Under review as a conference paper at ICLR 2022
frequency, can more skip-LSTMs of smaller skipping factors k. On the other hand, if the label changes are of low frequency, iec when the input sequence are sampled with relatively high frequency, can use more skip-LSTMs of larger skipping factors. One of important reasons for the excellent property of SegTima to handle time series of two scales is due to the MSS-LSTM net, since it has two important functions: (1) automatic adaption of the weights to focus on the more important frequency scale of input-output sequence pairs; (2) representation learning of temporal dependencies within the data.
The encoder-decoder structure (Sutskever et al. 2014) is a proven approach encoder-decoder (Figure with the two convolutional layers as the decoder (inspired by Chen et al. (2018). From the encoder two levels of features come out to the decoder: (1) low-level features from the early layers of ID-DS- ResNet; (2) the output features after the AMSP. These two levels of features exploit the multi-scale information and thus contribute to the property of Seg of handling multi-scale frequencies of data. Note that the encoder has a very deep architecture (in ID-DS- ResNett while the decoder has only several layers. This is because this structure can learn the representation efficiently and compresses them a bottleneck layer.
ID Encoder- Decooerr
in various fields and has great success the 2D problem semantic segmentation. We adopt a ID
and AMSP (explained later) as the encoder and
ID Depthwise Separable and Atrous Convolu- tion. A ID depthwise separable (DS) convolu- tion (Howard al., (017) separates ID normal convolution to Iwo steps (Appendix Figure [ and c): first depthwise convolution that computes in each channel independently; the a pointwise convo- lution that aggregates the different channels together. Atrous convolution (a.k.a. dilated convolution, Yu & Koltun 2016)) one computes dilated convolution with skipped values (Appendix Figure J...
Input
1D-Comw
conv.
output (Ixe)
Average pooling
stepwise softmax layer
iD-DS-ResNet. We adopt a ID-ResNet with depth- wise separable convolutions
part of the en- coder (Figure B3). In the bi-pass architecture, we have used LSTM in the other "pass". Thus, would like to rely on CNN here. The receptive fields in CNN usually of limited size. Thus they often can only model local temporal patterns. To model long-term "memory" LSTM does, normally more layers are required. The ResNet (He et al., 2016) is an established convolutional networks in image processing, and has been often used as module for semantic segmentation. More importantly, ResNet can provide very deep layers of CNN, effectively mitigating the gradient vanishing problem in deep architectures.
Figure 4: The stepwise segmentation. 1D- Conv: ID convolutional layer, 1: length of the sequence, c: cardinality of the labels.
Atrous Multi-Scale Pooling (AMSP). The AMSP (Figure E. inspired by Chen et al. (2018) fol- lows the same idea of handling information with multiple scales of frequency as that is in MSS- I.STM. In AMSP, a series of sequences (Figure 33)) that are down-sampled at different scales by dilation are concatenated and processed by a further ID-convolutional layer. The output sequence serves as the high-level features, together with the low-level features, are fed into different compo- nents in the decoder to help the network automatically adapt to the unknown frequency in data.
Stepwise Segmentation. We adopt a ID convolution layer followed by a an average pooling layer and softmax layer. The number of neurons of the 1D convolution layer is equal to the cardinality of output labels, whose outputs can be seen to represent the latent labels. Note the pooling factor of the average pooling layer is dependent the label change in HAR can be as fast as one label for 20 time steps, and thus the pooling factor is set to 1, namely no pooling is performed. On the other hand, the label in sleep staging is assigned to every 3000 time steps, so the pooling factor can be 3000.
the label change frequeney in the output. For example,
In summary, SegTime has several advantages: (1) it can handle time series segmentation in two scales, fast and slow changing labels, thanks to the multi-scale structures in MSS-LSTM, ID- encoder-decoder, and AMSP: (2) SegTime reduces parameters and computation via multiple mech- anisms of depthwise separable convolution, atrous convolution, and Skip-LSTMs; (3) it incorporates
5

---

Under review as a conference paper at ICLR 2022
the long-term dependencies via MSS-LSTM and very deep architecture of -et-Ret (4) it allows a precise label prediction at step-level, which is due to the previous points and the stepwise segmentation module.
EVALUATION
This section first describes the datasets, then explains the experiment design, and present and dis- cusses the results. We evaluated SegTime extensively on public datasets covering two time scales: both fast and slow changing labels. On each time scale, SegTime beats their corresponding state-of- the-art baselines, at least achieves comparable performance.
4.1 DATA DESCRIPTION
Data statistics. Average segment length is the to- tal number of time steps divided by the total number of seg- ments. The smaller is, the faster the label changes. Minimal segment length is the length the shortest segment.
Table I:
An overview of the data statistics is given in Table T It can be seen the Opportunity Drill datasets have relatively fast changing labels, and the Sleep-EDF datasets have relatively slow changing labels (More de- tails see Appendix Table and 8).
Dataset
#Time steps
Sampling rate
Average segment length
Minimal segment length
Opportunity Drill Sleep-EDF-39
114257
30Hz
102
Opportunity Drill. The opportunity dataset is a popular variation of Human Activity Recognition Dataset (Roggen et al. 2010 C Chavarriaga et al.
123723k
100Hz
26045
3000
2013) extensively studied
for the task of HAR (Nweke et al. 2018] Yao et al. 2018] Gjoreski et al.
(2166). contains of 243
time series measured from wearable, object, and ambient sensors with frequency of 30 Hz, includ- ing accelerometers, location sensors, switches, inertial measurement units, etc. We have chosen 113 sensors among them common practice in the literature (Roggen al. 2010 Chambers & Yoder. 2020). The dataset has two groups of data, the activity of daily life (ADL) set and the drill set. The former one is recorded when the subjects are doing normal daily life activities in the morning, and the it has slower changing label (average segment length 194), while the latter one is recorded dur- ing series of designed experiments, where the activities change intensively, thus resulting in faster label changes (average segment length 102). We select the opportunity drill set of subject 3 and 4, since many input signals of subject and 2 are missing. The dataset has 7 output label sequences of body locomotion (stand, lie, sit, walk), activities (relaxing, clean up, etc.), arm gestures (cut, clean, release, etc.) and objects in the arms (bottle, salami, bread, etc.). Among which, we choose locomo- tion, right arm gestures, aright arm objects. and both arm gestures for evaluation, because the other label sequences almost do not change, and thus reporting on them does not bring too much. In all label features, there exists a background class (i.e. null class) where no relevant action is performed. The dataset collected from 4 subjects, each contributed six subsets of different experiment runs. The time series lengths in these subsets range from about 25000 to 70000 time steps.
Sleep-EDE. This taken from a public PhysioNet datbase (Kemp et al. 2000 Goldberger et al. 2000) often used for benchmarking sleep staging algorithms. We choose the Sleep-EDF-39 dataset, sleep-cassette subset of 39 whole-night polysomnographic sleep recordings of healthy Causasions (taking no sleep-related medication) from age 25 to 101, because it is extensively studied in the literature (Perslev et al., 2019 2021 Samiee et al., 2015). In sleep staging, the continuous EEG categorised into stages like "Awake", "N1", "N2", "N3", and "REM". We follow the convention of sleep staging problem: we only consider the signals starting from 30min before 30min after the first and last nok-wake sleep stage; meanwhile we merge the sleep stages S3 and $4 into a single stage N3, according to the American Academy of Sleep Medicine characterisation (Conrad & AASM 2007).
and EOG signals segmented and the segments
4.2 EXPERIMENT DESIGN
We now explain our experiment design for evaluating SegTime. We conduct experiments on popular datasets with two time-scales: fast changing labels (Opportunity Drill) and slow changing labels (Sleep-EDF). We select one baseline for each of the time-scale. In addition, we adapt a state-of-the the art neural network for the 2D problem of semantic segmentation. Thus, there exist four architectures to test: three baselines and SegTime. We test the four architectures on the two dataset against three performance metrics, and obtain 8 models in total.

---

Under review as a conference paper at ICLR 2022
Baselines. Based on the datasets of the different time scales, we select CNN Yao et al. (2018) for the Opportumity dataset (HAR task) and U-Time (Perslev et al.] 2019) for the Sleep-EDF dataset (sleep staging task). In addition, we select a popular architecture for segmantic segmentation, the Deepl.ab (Chen et al. 2018), and adapt it to a ID version for the time series segmentation.
DM This is popular CNN-LSTM networks highly-optimised for the opportunity dataset (Ordóñez & Roggen 2016). It has four convolutional layers and two LSTM layers. The work (Ordóñez & Roggen (016) adopted the common sliding window approach and tested Deep- ConvLSTM with both ADL and drill datasets. We reimplement the DeepConvLSTNN and test on the drill datasets.
U-Time. U-Time is a feed-forward CNN-based encoder-decoder neural networks created based on U-Net (Ronneberger al.) 2015), designed to classify fixed length signals for sleep staging, U- Time has achieved state-of-the
performance on the Sleep-EDF dataset, according to to/Peslee et al.) (2019). follows the classic wawpiid-elassff-ccccaaeeeee
strategy, and is focused on datasets with two input signals and one sequence of output labels of low frequency change (i.e. high sampling rate for input signals, 100Hz- - 521Hz).
SegTime". This is a transfer and adaptation from Deeplaabv3+, which is a state-of-the-art approach for semantic segmentation (Chen et al.] 2018). also adopts a CNN-based encoder-decoder ar- chitecture. In addition, it merges the depthwise separable convolution with atrous spatial pyramid pooling, and thus can exploit multi-scale information. DeepLaby3+ was not defined for time series segmentation, but we consider it very promising. Thus, we adapt its third version to ID architec- ture, adding a stepwise segmentation module with average pooling for merging outputs. We use it as one of our baselines. We name it as SegTime" since does not have the MSS-LSTM module.
Performance Metrics. We have selected three performance metrics: accuracy, weighted F-score, weighted loU to evaluate the stepwise prediction of output label sequences.
Accuracy (Ace). Suppose the ground-truth label sequence is y = [y,,..., VL], and the predicted [21, 22, ... ZL], where 22,...,
label sequence is 9 [6:: 122, ..., Dr.). Construct the evaluation sequence
2, = 1 y/ Gz. otherwise 2, = O. The Ace for each single label class is: (TP + TN)) + N).
Weighted F-score (Fw)) We use this metric previous work (Roggen et al. 2010 Chambers &
Yoder 2020 have used it. For each label class, the F-score is calculated as T TP/(TP + + FNJ]. where TP refers to true positive prediction, FP refers to false positive prediction, and FN refers to false negative prediction. The Weighted F-score F.e is an average of the F-score of each label class. For the class c, the number of labels the ground-truth label sequence that belong to e is indicated as N₂. Thus the sum of all number of labels is L, i.e. L. = L. No. Weighted F-score is then calculated the weighted average of F-scores of all label classes as: F.. I E. N.F Sscore.
Weighted loU (ToUw.) Weighted Mean of Intersection over Union ToU is a common evaluation metric for semantic image segmentation (Long et al., 2015). We introduce it for time series segmen- tation. For each label class, the loU is calculated as TP/(P + FN + FP), the weighted IoU is then calculated as loU', T. >. NoloU
Cross Validation and baselines and SegTime the datasets. On the Sleep-EDF datasets, we follow the convention in the literature (Perslev et al.] (201)): to split the data on a per-subject basis, namely to put the samples belong to the same test subject to the same train/validation/eesst set. We selected the hyper-parameters based on the model performance on the validation set. We set the maximum training epoch as 200, and adopt early-stopping training strategy, when the loss on the validation set does not decrease any more for consecutive 20 epochs. The hyper-parameter selection follows a mixed scheme of limited grid search (varying one and fixing others) and human heuristics.
Selection. We conducted 20-fold cross validation for the
Optimisation and Implementation Details. Since the number of labels are highly imbalanced (Ap- pendix Table 8 and 9) we adopted a on-the-fly training scheme to counter the bass-imbalaane issue: for each epoch, we first define the percentage P. (the chance of being sampled) for each class, and then we sample K sub-sequence in total during the random sampling process, with at least K Pa beb-sequences contains the class c. Further more, we randomly add global and regional Gaussian noise to further increase the robustness of the methods. The source code is attached in the supple- mentary materials. The code is based on Python (Van Rossum & Drakel 2009) and Pytorch (Paszke) et al. 2019). We ran all experiments on Amazon SageMaker with NVIDIA Tesla K80 GPU clusters.
7

---

Under review as a conference paper at ICLR 2022
Table 2:
Results on the Opportunity Drill dataset (HAR). Loc.: locomotion, Ges.(R): gesture of the right
arm, Obj.(R): object in the right arm, (Bs. gesture of both arms, Mean.: weighted mean value of the corresponding metric.
Methods Metrics
Opportunity Drill with Background
Opportunity Drill without Background
Loc.
Ges.(R) (kj.(R) (Bs.(B) Mean
Loc.
Ges.(R) Ohj.(R) Ges.(B)
Mean.
Deep-
Ace
0.84
0.86
0.87
0.88
0.86
0.91
0.81
0.83
0.83
0.85
Conv-
F"
0.80
0.86
0.87
0.87
0.85
0.91
0.84
0.87
0.88
0.88
LSTM
ToU
0.72
0.77
0.78
0.78
0.76
0.84
0.73
0.79
0.79
0.79
Acc
0.89
0.83
0.93
0.79
0.86
0.89
0.75
0,92
0.68
0.81
U-Time
Fw
0.89
0.83
0.93
0.78
0.86
0.90
0.78
0.95
0.70
0.83
IoU,.
0.80
0.73
0.87
0.68
0.77
0.81
0.66
0,91
0.59
0.74
Ace
0.90
0.85
0.90
0.87
0.88
0.89
0.77
0.87
0.84
0.84
SegTime"
Fi. 0.90
0.85
0,90
0.87
0.88
0.90
0.80
0,91
0.87
0.87
IoU...
0.81
0.74
0.82
0.77
0.79
0.81
0.68
0.84
0.79
0.78
Acc
0.89
0.88
0.88
0.89
0.88
0.88
0.81
0.85
0.86
0.85
SegTime
Fw 0.88
0.87
0.88
0.89
0.88
0.89
0.84
0.89
0.91
0.88
IoU
0.80
0.78
0.80
0.81
0.80
0.80
0.74
0.82
0.84
0.80
We used the Adam optimiser (Kingma & Ba 2019) with an adaptive learning rate strategy to reduce the learning rate by factor of 10 if the validation loss stops improving for more than onsecutive 20 epochs (Note that this is easily done with CeduceLRRnnPlateal in Pytorch). We minimise the cross-entropy cost function with Kaiming class weights initialisation (He et al.. (015). We adopt batch size of 32 for the opportunity dataset and 128 for the Sleep- EDF dataset.
4.3 RESULTS AND DISCUSSION
We present and discuss the results on datasets of two time scales, and additionally the ablation study.
Time-Scale of Fast Changing Label. The results of four architectures tested on the dataset with labels of high frequency change (Opportunity Drill) are shown in Table [ The opportunity drill dataset has six output label sequences: (1) locomotion, gesture of the (2) right arm and (3) left arm, object in the (4) right arm and (5) left arm, (6) gesture of both arms. Among which, the output labels of (3) and (5) almost do not change, and thus reporting on them does not bring too much. We there- fore leave them out, Important to note, the opportunity drill labels of background class (namely no activity label is assigned to the time steps). Ideally, good architecture should not learn the dataset particularity, and be insensitive to the background class. Thus, it is also important angle whether the network is insensitive to exclusion of background class in performance calculation.
is very unbalanced, and has many
We first compare the model when the background class is considered in the performance calculation. By looking at rows of accuracy and the column of Mean (average weighted), it can be easily seen that SegTime outperforms SegTime by about 1,, and SegTime significantly outperforms U-Time and CNN by 3% or more. By comparing the performance on the four output features, we see although SegTTime is not always the best, it achieves consistent good performance. By looking the output feature of Ges.(B), we see that the performance of U- U-Time drops drastically. We postulate the reason is that, U-Time tailored more slower changing labels, while the label of Ges. changes very fast, and thus the performance of U-Time degrades significantly. The weighted Fl-score and weighted IoU are in accordance with Acc, and also demonstrate that SegTime can outperform the ttate-of-the-an baselines for dataset with labels of fast frequency change.
We then look at the models with background class excluded from the performance calculation. We still first look and the rows of accuracy and the column of Mean,,, which shows all models have some deterioration in performance. Yet, SegTime still outperforms all other baselines and has relatively small deterioration, which means Seg Time is more insensitive to the background class, compared to U-Time, whose performance is worsened by about 4% when the background class is removed from calculation. If we look the accuracy of the four output features, SegTime still has consistent good results, while the other baselines have drastic degradation on one or more output feature.
Time-Scale of Slow Changing Label. The results of four architectures tested the Sleep-EDF datasets are shown in Table 3 We look at the rows of F,,, and the column of Mean,.. It can be seen that the two variants of SegTime achieve comparable results to the most optimised baseline U-Time.
8

---

Under review as a conference paper at ICLR 2022
of sleep staging. W stands for the awake stage, REM stands for rapid eye movement stage, N1, N2 and N3 stand for the three stages of non- non-REM sleep, and Mean. stands for weighted mean value of the corresponding metric. The results of U-meme is directly taken from the baseline reported in Perslev al. 2019). which has valid significant digits.
Table Results on the Sleep-EDD dataset
Dataset
Model
Metrics
W
NI
REM
Mean,..
Deep-
Ace
0.28
0.00
0.91
0.17
0.14
0.51
Conv-
Fw
0.41
0.00
0.68
0.21
0.19
0.44
LSTM
IoU
0.26
0.00
0.51
0.12
0.10
0.31
Ace
0.90
0.46
0.88
0.83
0.84
0.83
U-Time
Fw 0.87
0.52
0.86
0.84
0.84
0.82
Sleep- EDF- 39
IoU'
0.77
0.35
0.76
0.73
0.72
0.71
Acc
0.35
0.84
0.92
0.89
0.83
0.82
SegTime
Fw
0.87
0.40
0.89
0.82
0.77
0.82
IoU
0.77
0.25
0.79
0.70
0.63
0.71
Acc
0.31
0.89
0.91
0.86
0.76
0.82
SegTime
Fw
0.82
0.38
0.88
0.81
0.81
0.82
IoU
0.70
0.24
0.78
0.68
0.69
0.70
Looking the detailed accuracy on the sleep staging labels, we see that the baselines and SegTime perform differently well on the labels. Note that both U-Time and SegTime" are adapted from highly optimised and sophisticated architectures for semantic segmentation, which have achieved outstanding performance in the highly competitive domain of semantic segmentation. The baseline DeepConvLSTM is tailored to the opportunity dataset with fast changing labels. Its architecture not deep enough model the long-term dependencies. It is thus not surprising to see a drastic degradation of its performance on Sleep-EDD datasets. The work [eesllev et al. (2019) used Mean to compare the performance, but consider the Mean,, more appropriate than the simple mean values for performance comparison, since the labels are highly unbalanced.
Ablation Study. We conduct ablation study test the contribution of the two major modules SegTime that comprise the bi-pass architecture: the MSS-LSTM and the ID-encoder-decodde Ta- ble 4 shows the results. The column w/o MSS- LSTM is SegTime without the MSS-LSTM net module, it namely the SegTime". The column w/o -D-encoder-deccoder is Time without the ID-encode-decoder module. can be seem that both modules can increase the performance of SegTime. The SegTime" already achieves very good performance, while the MSS-LSTM module further improves the performance by about 1%.
Table 4: Ablation study. Two major modules of Seg- Time (MSS-LSTM and ID-encoeer-eecader) are in- dependently tested. w/o: without.
Dataset
Metries
w/o
w/o ID-
SegTime
MSS- LSTM
encoder- decoder
Opportunity Drill
Acc
87.75%
86.58%
88.43%
F..
0.8763
0.8653
0.8831
5 CONCLUSION
In this work, we propose a novel neural network architecture, SegTime, that allows to do time series segmentation for datasets of wider spectrum of time scales. We conducted experiments of comparing SegTime and highly-optimised baselines on datasets with fast changing and slow changing labels. The results show that Seme outperform the baselines comparable performance on both time scales. SegTime has multiple architectural advantages: bi-pass architecture that combines LSTM and very deep CNN-based ID-ensoder-decoder, several multi-scale structures, depthwise separable and atrous convolution, and a stepwise segmentation module. We are very excited about our approach and its results: besides promising insights its opens exciting avenues for further research and practical industrial solutions.
least achieve

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
To reproduce our results, we provide the following resources:
1. The source code, including the SegTime model details, layers, hyper-parameters, etc., link and description of the datasets in the supplementary materials.
2. Detailed model topology, hyper-parameters of the models, data descriptions, and model illustra- tions. In the Appendix, Table I and give a detailed view of the ID-DS-ResNet module. Ta- gives detailed view of the AMSP module, and 5 detailed view of the Decoder module, T gives a detailed view of the MSS-LSTM Net module for the SegTime evaluated on the Sleep-EDF set. Table 8 and | give more detailed explanation of the output features and labels for the opportunity dataset and Sleep-EDF dataset. Figure [ schematically illustrates the ID depthwise separable and atrous convolution. Figure 2 gives expanded overview of the SegTime architecture.
3. Our amonymous code repository:
https: hattps://anonymouss http://wwww....ooooooooee/e////s 0546/
REFERENCES
Andreas Bartschat, Markus Reischl, and Ralf Mikut. Data mining tools. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4):e 2019.
Atanu Chakraborty, Debojoyti Mukherjee, Amit Dutta, Aruna Chakraborty, and Dipak Kumar Kole. An effective stock price prediction technique using hybrid adaptive neuro fuzzy inference system based on grid partitioning. In Intelligent Systems Technologies and Applications, pp. 275-284. Springer, 2016.
Robert D Chambers and Nathanael C Yoder. Filternet: A many-to-mann deep learning architecture for time series classification. Sensors, 20(9):2498, 2020.
Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. The opportunity challenge: benchmark database for on-body censor-based activity recognition. Pattern Recognition Letters, 34(15):0033-2042 2013.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018
Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang. FasterSeg: Searching for faster real-time semantic segmentation. In International Conference on Learning Representations, 2019.
Iber Conrad and AASM. The AASM manual for the scoring of sleep and associated events: rules, terminology and technical specifications. American Academy of Sleep Medicine, Westchester, IL, 2007.
Philippe Esling and Carlos Agon. Time-series data mining. ACM Comput. Surv, 45(1), December 2012.
Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi. Kenji Nagamatsu, and Shinji Watanabe, End-to- end neural speaker diarization with permutation-free objectives. In INTERSPEECH, 2019.
André Gensler and Bernhard Sick. Novel criteria to measure performance of time series segmenta- tion techniques. In LWA, pp. 193-204. Citeseer, 2014.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning forget: Continual prediction with Istm. Neural computation, 1 2210):2451-2711 2000.
Hristijan Gjoreski, Jani Bizjak, Martin Gjoreski, and Matjaž Gams. Comparing deep and classical machine learning methods for human activity recognition using wrist accelerometer. In Proceed- ings of the IJCAI 2016 Workshop on Deep Learning for Artificial Intelligence, New York, NY, USA, volume 10, pp. 970, 2016.
10

---

Under review as a conference paper at ICLR 2022
Ary L Goldberger. Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George Moody, Chung-Kang Peng, and Eugene Stanley. Phys- ioBank, ToolioToolkit, and PhysioNet: components of new research resource for complex physiologie signals. circulation, 101(23)=211--2200 2000.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 704.048611, 2017.
ITU. Recommendation ITU-T Y.2060: Overview of the Internet of things. Technical report, Inter- national Telecommunication Union, 2012.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the EEG. IEEE Transactions on Biomedical Engineering, 47(9):1185-1194, 2000.
Eamonn Keogh, Selina Chu, David Hart, and Michael Pazzani. An online algorithm for segmenting time series. In Proceedings 2001 IEEE international conference on data mining, pp. 289-296. IEFE, 2001.
Eamonn Keogh, Selina Chu, David Hart, and Michael Pazzani. Segmenting time series: A survey and novel approach. In Data mining in time series databases, pp. 1-21. World Scientific, 2004.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 95-104, 2018.
Wei-Han Lee, Jorge Ortiz, Bongjun Ko, and Ruby Lee. Time series segmentation through automatic feature learning. arXiv preprint 1801v:1801.055: 2018.
Zidong Li. Shuai Wang, Shuang Song, Yaping Wang, and Walter Musakwa. Detecting land degra- dation in southern africa using time series segment and residual trend (tss-restrend). Journal of Arid Environments, 184:104314 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-344000 2015.
Philipp Meschenmoser, Juri F Buchmüller, Daniel Seebacher, Martin Wikelski, and Daniel A Keim. MultiSeg VA: Using visual analytics to segment biologging time series on multiple scales. IEEE Transactions on Visualization and Computer Graphics, 27(2):162311633 2020.
Henry Friday Nweke, Ying Wah Teh, Mohammed Ali Al-Garadi, and Uzoma Rita Alo. Deep learn- ing algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Systems with Applications, 105:233-261, 2018.
Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1): 15, 2016.

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Mathias Perslev, Michael Hejselbak Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian Igel. U-time: a fully convolutional network for time series segmentation applied to sleep staging. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pp-
4415-4426, 2019.
Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul Jørgen Jennum, and Christian Igel. U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine, 4(1):1-12, 2021.
Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS), pp. 243-240. IEEE, 2010.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Kaveh Samiee, Péter Kovács, Serkan Kiranyaz, Moncef Gabbouj, and Tapio Saramaki. Sleep stage classification using sparse rational decomposition of single channel EEG records. In 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 1860-1864. IEEE, 2015.
Joan Serra, Meinard Müller, Peter Grosche, and Josep LI Arcos. Unsupervised music structure anno- tation by time series structure features and segment similarity. IEEE Transactions on Multimedia, 16(5):1229 1200022222220, 2014.
Quanxi Shao, Zhanling Li, and Zongxue Xu. Trend detection in hydrological time series by segment regression with application to shiyang river basin. Stochastic Environmental Research and Risk 2010.
Assessment, 24(2):221
Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(1 1): 1998-200000, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112 3112, 2014.
Guido Van Rossum and Fred L. Drake. Python Reference Manual. CreateSpace, Scotts Valley, CA, 2009, ISBN 1441412697.
S. Waczowicz, M. Reischl, V. Hagenmeyer, R. Mikut, S. Klaiber, P. Bretschneider, I. Konotop, and D. Westermann. Demand response clustering how do dynamic prices affect household electricity consumption? In Proc., IEEE PowerTech, Eindhoven, pp. 1-6. IEEE, 2015. doi: 10. 10.11099phc2201572224499
Huaijun Wang. Jing Zhao, Junhuai Li, Ling Tian, Pengjia Tu, Ting Cao, Yang An, Kan Wang. and Shancang Li. Wearable sensor-based human activity recognition using hybrid deep learning techniques. Security and Communication Networks, 2020, 2020.
Quan Wang. Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopz Moreno. Speaker Diarization with LSTM. In 20/8 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5239-5243. IEEE, 2018.
S. Wolf, T. Loose, M. Schablowski, L. Döderlein, R. Rupp, H. J. Gemner, G. Bretthauer, and R. Mikut. Automated feature assessment in instrumented gait analysis. Gait & Posture, 23(3): 331-338, 2006.
12

---

Under review as a conference paper at ICLR 2022
Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In founth international joint conference artificial intelligence, 2015.
Qiang Yang and Xindong Wu. 10 challenging problems in data mining research. International Journal of Information Technology & Decision Making, 05(04))597-604, 2006.
Rui Yao, Guosheng Lin, Qinfeng Shi, and Damith C Ranasinghe. Efficient dense labelling of human activity sequences from wearables using fully convolutional networks. Pattern Recognition, 78: 252-266, 2018.
Jiangling Yin, Yain-Whar Si, and Zhiguo Gong. Financial time series segmentation based turning points. Proceedings 2011 International Conference on System Science and Engineering, pp- 394 399. IEEE, 2011.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- tional Conference on Learning Representations, 2016.
Aonan Zhang, Quan Wang. Zhenyao Zhu, John Paisley, and Chong Wang. Fully supervised speaker diarization. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6301-6305. IEEE, 2019.
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
Appendix Table I: SegTime model topology Module 1D-DS-Resse Tige for the Sleep-EDD dataset,
Part l
Layer (type)
Param Output Shape
Convld-1
[-1, 2, 52500]
katchNormld
[-1, 2, 52500]
4
ReLU-3
[-1, 2, 52500]
0
MaxPoolld-4
[-1, 2, 26250]
0
Convld-5
[-1, 2, 26250]
4
BatchNormld-6
[-1, 2, 26250]
4
ReLU-7
[-1, 2, 26250]
0
Convld-8
[-1, 2, 26250]
12
BatchNormid-9
[-1, 2, 26250]
ReLU-10
[-1, 2, 26250]
Convld-11
[-1, 4, 26250]
8
BatchNermld-12
[-1, 4, 26250]
8
Convid-13
[-1, 4, 26250]
BatchNormld-14
1-1, 4, 26250]
ReIU-15
1-1, 4, 26250]
1ottleneck1D-16
[-1, 4, 26250]
Convid-17
[-1, 2, 26250]
BatchNormld-18
[-1, 2, 26250]
ReLU-19
1-1, 2, 26250]
Convld-20
[-1, 2, 26250]
12
BatchNerm1d-21
[-1, 2, 26250]
4
RcLU-22
[-1, 2, 26250]
0
Convld-23
[-1, 4, 26250]
BatchNormld-24
[-1, 4, 26250]
8
ReLU-25
[-1, 4, 26250]
0
Bottleneck1D-26
[-1, 4, 26250]
0
Convid-27
[-1, 4, 26250]
BatchNorm1d-28
1-1, 4, 26250
ReIU-29
1-1, 4, 26250
0
Convld-30
1-1, 4, 13125
48
BatchNormld-31
[-1, 4, 13125]
8
ReLU-32
[-1, 4, 13125]
Convld-33
[-1, 8, 13125]
32
BatchNormld-34
[-1, 1, 13125]
16
Convid-35
[-1, 8, 13125]
32
BatchNermld-36
[-1, 8, 13125]
16
ReLU-37
[-1, 8, 13125]
BottleneckID-39
[-1, 8, 13125]
0
Convid-39
[-1, 4, 13125]
32
BatchNorm1dd4f
[-1, 4, 13125]
8
ReLU-41
[-1, 4, 13125]
do2v1d-42
[-1, 4, 13125]
48
BatchNormld-43
1-1, 4, 13125
8
ReLU-44
1-,, 4, 13125
0
Convld-45
1-,, 8, 13125
32
BatchNormld-46
[-1, 8, 13125]
ReLU-47
[-1, 8, 13125]
0
BottleneckID-48
[-1, 8, 13125]
0
Convid-49
[-1, 8, 13125]
64
BatchNormld-50
[-1, 8, 13125]
16
14

---

Under review as a conference paper at ICLR 2022
Appendix Table 2: SegTime model topology Module ID-DS-ResNet Time for the Sleep-EDD dataset,
Part II
Layer (type)
Output Shape
Param !
ReLU-51
[-1, 8, 13125]
Convld-52
[-1, 6, 13125]
192
Batchhormld-53
[-1, 8, 13125]
16
ReLU-54
[-1, 8, 13125]
0
Conv1d-55
[-1, 16, 13125]
128
BatchNormld-56
[-1, 16, 13125]
32
Convid-57
[-1, 16, 13125]
128
B5tchNorm1d-58
[-1, 16, 13125]
32
ReLU-59
[-1, 16, 13125]
0
BottJeneckID-60
-1, 16, 13125
Convld-61
1-1, 8, 13125
128
BatchNorm1d-62
1-1, 8, 13125
16
ReLU-63
[-1, 8, 13125]
0
Conv1d-64
[-1, 8, 13125]
192
BatchNormld-65
[-1, 8, 13125]
16
ReLU-66
[-1, 8, 13125]
0
Convld-67
[-1, 16, 13125]
128
BatchNermld-68
[-1, 16, 13125]
32
ReLU-69
[-1, 16, 13125]
Bottleneck1D-70
11,, 16, 13125]
0
Convid-71
1-1, 8, 13125
128
BatchNorm1d-72
[-1, 8, 13125]
16
Rel.U73
[-1, 8, 13125]
Convld-74
[-1, 8, 13125]
192
BatchNermld-7?
[-1, 8, 13125]
16
ReLU-76
[-1, 8, 13125]
0
Convld-77
[-1, 16, 13125
128
BatchNormld-78
[-1, 16, 13125]
32
ReLU-79
[ [-1, 16, 13125]
0
Bot
Bottleneck1D-80
1, 16, 13125]
0
Convld-81
[-1, 8, 13125]
128
BatchNormld-62
[-1, 8, 13125]
16
ReLU-83
[-1, 8, 13125]
0
Convld-84
[-1, 8, 13125]
192
BatchNerm1d-85
1-1, 8, 13125]
16
ReLU-86
1-1, 8, 13125
0
Conv1d-87
1 1-1, 16, 13125
128
BatchNormldd-B
[-1, 16, 13125]
32
ReLU-89
[ [-1, 16, 13125]
0
BottleneckID-90
[-1, 16, 13125]
0
Convld-91
[-1, 16, 13125]
256
BatchNorm1d-92
[-1, 16, 13125]
32
ReLU-93
[-1, 16, 13125]
0
Convld-94
[-1, 16, 13125]
768
BatchNormld-95
[-1, 16, 13125]
32
ReLU-96
[-1, 16, 13125]
Convld-97
[-1, 32, 13125]
512
BatchNerm d d-98
[-1, 32, 13125]
64
Convld-99
[-1, 32, 13125]
512
catchhormld-100
-1, 32, 13125
64
15

---

Under review as a conference paper at ICLR 2022
Appendix Table 3: SegTime model topology Module 1D-DS- Ressett of Tigeime for the Sleep-EDF dataset, Part III
Layer (type)
Param Output Shape
ReLU-101
[-1, 32, 13125]
0
BottleneckID-DD
[-1, 32, 13125]
0
Convld-103
[-1, 16, 13125]
512
Batchhormld-104
[-1, 16, 13125]
32
ReLU-105
[ 1-1, 16, 13125]
0
Convid-106
[-1, 16, 13125]
768
BatchNormlld-00
[-1, 16, 13125]
32
ReLU-108
[-1, 16, 13125]
0
Cenvld-109
1-1, 32, 13125]
512
BatchNormld-110
[ [-1, 32, 13125]
64
ReLU-111
[-1, 32, 13125]
0
BottleneckID-112
[-1, 32, 13125]
0
Convid-113
[-1, 16, 13125]
512
BatchNormld-114
[-1, 16, 13125]
32
ReLU-115
[-1, 16, 13125]
0
Convld-116
[-1, 16, 13125]
768
BatchNormld-117
[-1, 16, 13125]
32
ReLU-118
1-,, 16, 13125
Convld-119
1 -1, 32, 13125
512
BatchNerm1d-120
1-1, 32, 13125
64
ReLU-121
[-1, 32, 13125]
0
BottlenecklD-122
[-1, 32, 13125]
0
Total params: 8, 662
Trainable params: 8, 662
Non-trainable params:
Input sizc (MB) : 0.40
Forward/backward pass size (MB) 151. 41
Params size (MB) 0.03
Rstimated Total Size (MB) 151.84
input
kernel
depthwise kernel
pointwise kemee
atrous kernel
output
1-d normal conv.
1-d depthwise conv.
1-d pointwise conv.
1-d atrous conv.
Appendix Figure 1: Depthwise separable convolution (conv.) and atrous convolution.
16

---

Under review as a conference paper at ICLR 2022
Appendix Table 4: SegTime model topology Module AMSP of Seme for the Sleep-EDF dataset
Layer (type)
Output Shape
Param
Convid-1
[-1, B, 13125]
256
BatchNormid-2
[-1, 13125]
16
ReLU-3
[-1, 8, 13125]
0
AMSPModule-4
[-1, 13125]
0
Convld-5
[-1, B, 13125]
768
BatchNormld-6
[-1, 8, 13125]
16
ReLU-7
[-1, 8, 13125]
0
AMSPModule-8
[-1, 8, 13125]
Convld-9
[-1, 8, 13125]
768
BatchNormld-10
[-1, 8, 13125]
16
ReLU-11
[-1, 8, 13125]
0
AMSPModule-12
[-1, B, 13125]
0
Convld-13
[-1, 13125]
768
BatchNorm1d-14
[-1, 8, 13125]
16
Re1U-15
[-1, 8, 13125]
0
AMSPModule-16
[-1, 8, 13125]
0
xdaptivekvgFoolld-11
[-1, 32, 1]
Convid-18
[-1, 8, 1]
256
ReLU-19
[-1, 8, 1]
Convld-20
[-1, 8, 13125]
320
BatchNorm1d-21
[-1, 8, 13125]
16
ReLU-22
[-1, 8, 13125]
0
Dropout-23
[-1, 8, 13125]
0
Total params: 3, 216
Trainable params: 3,216
Non-trainable params: 0
Input size (MB): : 1.60
Porward/backward pass size (MB) 16.02 02
Params size (MB) 0.01
Estimated Total Size (MB) 17.64
17

---

Under review as a conference paper at ICLR 2022
Appendix Table 5: SegTime model topology Module Decoder of SegTime for the Sleep-EDF dataset
Layer (type)
Output Shape
Param
Convid-1
[-1, 1, 26250]
BatchNormld-22
[-1, 26250]
2
ReLU-3
[-1, 1, 26250]
0
Convld-4
[-1, 26250]
6
BatchNormid-5
[-1, B, 26250]
16
ReLU-6
[-1, B, 26250]
Dropout-7
[-1, 8, 26250]
0
Convld-8
[-1, 8, 26250]
1 192
BatchNormles d-9
[-1, 8, 26250]
16
ReLU-10
[-1, 8, 26250]
0
Dropout-11
[-1, 8, 26250]
0
Total params: 446
Trainable params: 446
Non-trainable params: 0
Input size (MB) 42057.00
Forward/backward pass size (MB) 13.42
Params size (MB) : 0.00
Estimated Total Size (MB) 42070 46
Appendix Table 6: SegTime model topology Module MSS-LSTM of SegTime the Sleep-EDF dataset
Layer (type)
Output Shape
P ram
#
SkipLSTM-1
[-1, 3500, 2]
40
SkipLSTM-2
[-1, 2100, 2]
40
SkipLSTM-3
[-1, 1050, 2]
40
SkipLLTM-4
[-1, 525, 2]
40
Dropout-5
[-1, 105000, 2]
Total params: 160
Total params: 160
Non-trainable params:
18

---

Under review as a conference paper at ICLR 2022
Appendix Table 7: Hlpperpparmmtees of SegTime for the opportunity dataset Sleep-EDF dataset.
Parameter
Value
Comments
Loss function
Cross entropy
A classic loss function for classficiation.
Regularisation
None
Class balancing
Customised
We tried to balance different classes, but not everyone, es- peically for those that under-represented in the datasets.
Optimiser
Adam
Kingma Ba (2019)
Initial learning rate
le 3e
We adopt adaptive learning rate using RedlaeL..OnPlateau in Pytorch. le-3 is the initial learning rate for the opportunity dataset, and 5e-1 for the Sleep-EDF dataset.
81
0,9
82
0,999 le
Prediction resolution
3000
I for opportunity, 3000 for Sleep-EDF, This is determined by the average pooling in the stepwise segmentation mod- ule.
Input dimension Input sequence length
113,
113 for the opportunity dataset, for the Sleep- EDF dataset
600, 3503000
600 for the opportunity dataset, 3543000 for the Sleep-EDF dataset
Conv. kernel size
1,3
There exist various kernel size, please refer to the detailed
model topology
Conv. kernel dilation size
1, 12, 24, 36
Multi-scale dilated (atrous) convolution
Max-pooling kernel size
3
seerage-pooligg kernal size
1, 3000
I for the opportunity dataset because it has fast changing label, 3000 for the Sleep-EDF dataset because it
Padding Up-sampling
Same
Nearest neigh-
bour
Activations
ReLU
Back normalisation Parameters
22m, 12k
22 million for the opportunity dataset, 12k for the Sleop- EDF dataset
Normalisation
Min-max, robust 128,
Min-max for the opportunity dataset, robust normalisation for the Sleep-EDF dataset.
Batch size
for the opportunity dataset, 32 for the Sleep-EDF dataset.
Early stopping criteria
Validation Ac- curacy
Accuracy computed overl all time steps
Model selection criteria
Validation Ac-
We set the maximum training epoch 200, and adopt early-stopping training strategy, when the loss the vali- dation dataset does not decrease any more for consecutive 20 epochs.
curacy
Training epochs Examples per epoch
200 1800, 800
1800 for the opportunity dataset, 800 for the Sleep-EDF dataset
19

---

Under review as a conference paper at ICLR 2022
Appendix Table 8: Details of the opportunity dataset
Output fea- ture name
Encoding
Description
Label names
Locomotion
Loc.
The action move- ment the subject.
Stand, walk, lic
Right
Ges.
The gesture of the right arm of the subject. The object held in the right hand of the sub- ject.
Unlock, stir, lock, close, reach, open, sip, clean, bite, cut, spread, release, move
gesture
Right arm
Ob).(R
Bottle, salami, bread, sugar, dishwasher, switch, milk, drawer3 (lower), spoon, knife cheese, drawer2 (middle), table, glass, cheese, chair, doorl, door2, plate, drawerl (top), fridge, cup, knife salami, lazychair
object
Both arms gesture
Ges.(B)
The gesture the both
Open door, close door, open fridge, close fridge, open dishwasher, close dishwasher, open drawer, close drawer, clean table, drink from cup, toggle switch
arms of the subject.
Appendix Table 9: Details of the Sleep-EDF dataset. The Sleep-EDF-39 dataset shorten as "39" and the Sleep-EDF-153 dataset is shortened as "153".
Label name Encoding Description
Wake
W
The condition when the subject is awake or drowsy. The brain wave least 50% alpha waves. Short and light sleep stage. The brain wave is dominated by theta waves.
Non-REM1
N1
Non-REM2
N2
The brain wave theta waves and intercepted by phe- nomena named sleep spindles.
Non-REM3
N3
The brain wave is theta waves with high amplitude. Rapid eye movement occurs, The brain wave is both theta waves and alpha waves.
REM
R
20

---

Under review as a conference paper at ICLR 2022
input W
M
input sequence
Input sequence
Encoder
Decoder
1D-DS- 1D-DS-Real ReaNe
1D-Conv
Skip-LSTM
Skpp-LSTM K-2
Upsampling
Skip-LSTM
Skip-LSTM K=8
Upeampling
Concatenation
Upsampling
Upsampling
1D-Conv
Concatenation
Concatenation
AMSP
1D-Conv
Upsampling
MSS-LSTM Net
Dropout
output sequence
Concatenation
Stepwise Segmentation
1D-Conv
conv.
output
(Ixc)
Average pooling
stepwise softmax
layer
label
Appendix Figure 2: Expanded overview of the SegTime architecture.
21


Reviews:
[{"review_id": "_NlMXa0c4s0", "reviewer": "Reviewer_1Qsp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n* The paper addresses an important problem of time series, i.e. accurate segmentation of a time series \n* The philosophy of the presented approach (stepwise classification) seems promising\n* Using both LSTM and CNN to treat the multi-scale challenge of time series is interesting, especially how it is done in the paper with two separate modules.\n* The SOTA section is accurate and easy to read. \n* Having a single architecture able to deal with fast and low changing labels is clearly an important outcome of the paper.  \n\nNeutral:\n* While the architecture is novel (to the best of my knowledge), most of the used modules are adapted from existing literature.\n \nCons:\n* The technical section is very hard to read, the appendixes are not used at the best to explain deeper the architecture. The main problem is that there is no explanation of the role/the idea behind each module. What can achieve the MSS-LSTM that can not achieve the encoder-decoder ? Why use AMSP, 1D-DS-ResNet ... ? Looking especially at the ablation study, I wonder if the two modules are really useful and I can not judge because the evaluation is not sufficiently detailed (see below). \n* The experimental part is insufficient to judge the relevance of the proposed architecture:\n   * evaluating more datasets will be useful to assess the stability of the proposed approach wrt the SOTA. As it is, the results are very close and it is hard to assess the strength and weakness of the proposed approach wrt SOTA.\n   * the core thesis of the authors is to provide a model able to do stepwise segmentation and to retrieve accurately the bounds of each subsequence. Unfortunately, all results concern aggregated measures over the whole sequences like accuracy or F-score. It seems very important to have some results on the bounds found by the model compared to SOTA (maybe something like a delta-time between the ground truth and the inferred timestep ?). \n   * the datasets are not well detailed (for instance what is the proportion of each class ? the paper said that the problem is unbalanced but no further details). \n   * the results are not well detailed: for the ablation study, only one single aggregated result is reported, it is impossible to understand the real role of each component. \n   * there is no indication about the training time and the inference time of each model, and as the results are very close, it is important to know the computational cost of each model. \n", "summary_of_the_paper": "The paper presents a new deep architecture for segmentation of time series, i.e. to find the subsequences inside of a time series corresponding to different classes and to determine the bounds of those subsequences. The proposed architecture includes two core components: a) a kind of LSTM with skip connections to deal with the multi-scale problem of time series, b) an encoder-decoder module based on CNN and ResNet. The outputs of the two components are then used in a convolutional layer to provide a stepwise classification at each time step. The approach is evaluated on two classical datasets wrt 3 baselines.", "summary_of_the_review": "The approach seems interesting and promising but:\n* the technical part is too hard to follow and requires more explanation on the reason for the use of each module \n* the experimental part is incomplete missing at least an important experiment to show how well the proposed model answer to the problem of the accurate detection of the time series segments; and details on results to understand the ablation study. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HzhWEIg5bwI", "reviewer": "Reviewer_SjGH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper brings up many interesting questions and does interesting analyses. However, I have concerns primarily from the broad conceptual claims and missing background from literature in this area. \n\nA core conceptual insight in the paper is described in contribution #1. \n* There is a claim that most TS segmentation methods operate as \"divide-classify-concatenate,\" however, this isn't true. Most current papers in this area use Temporal Conversion Net architectures which (as the paper also mentions) typically operate on a sliding window approach. Much of this work until a couple years ago was using LSTMs, Conditional Random Fields, HMMs, or other temporal models which also did not operate under divide-classify-concatenate. The conceptual framework used by this paper is no different than the union of TCN an LSTM-based approaches (which has also been widely documented in the literature, e.g., in the Speech community).\n* There is also a claim that other works haven't looked at labels operating over multiple time-scales. This is reiterated on page 3: \"No prior work has investigated time series segmentation in two scales: labels of high frequency and low frequency change. Few works are dedicated a precise segmentation with step-level accuracy.\" These statements highlight that this paper is missing a large number of references from the computer vision and ML communities, often under the guise of fine-grained action recognition, working to overcome many of these same problems. The MS-TCN work below, for example, specifically addresses multi-scale issues like this. \n\nSome good references. I recommend doing a search for more in this direction. \n* Lea et al. \"Temporal Convolutional Networks for Action Segmentation and Detection\" CVPR 2017.\n* Bai et al. \"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling\" arxiv 2018.\n* Farha et al. \"Ms-tcn: Multi-stage temporal convolutional network for action segmentation\" CVPR 2019 and \"MS-TCN++\" at TPAMI 2020.\n* Kahatapitiya et al. \"Coarse-Fine Networks for Temporal Activity Detection in Videos\" CVPR 2021.\n\nRe: the stepwise segmentation module. On one hand I understand why you would want to reduce the temporal resolution of your output space (e.g., from input sampling rate of 3k hz to output of 30 hz). This is commonly done in areas like speech where acoustic models are often downsampled to word-fragments at 30 (or 100) hz. However, this seems antithetical to the premise of the paper, where the goal is to have very precise timestamps. \n\nRe: SegTime-. While I like the simplify of the multi-scale Skip LSTM idea, I appreciate the ablations between the full (TCN+LSTM) model and the adaptation of DeepLabv3 (TCN). While the LSTM-based model does improve performance, I wonder if making other modifications to the TCN-side of the model could have the same impact. Ablations on other TCN architectures could be useful here. It's unclear what issues the LSTM-side prevents which couldn't alternatively be done with a modified TCN architecture. \n", "summary_of_the_paper": "This paper focuses on time-series (TS) segmentation. They claim that in the typical approach for this problem -- where you is to apply a model (e.g., a temporal conv net) over fixed windows in time in sliding window fashion -- it is challenging to predict precise breakpoints, especially when the labels change frequently relative to the sampling rate of the input data/sensors. They also claim that these approaches ignore long-term dependences. They introduce a network which they \"obviates\" the need for sliding windows and can precisely find breakpoints.\n\nTheir claimed contributions are: \n* A conceptual framework for TS segmentation\n* An architecture for solving TS segmentation problems\n* An adaptation of DeepLabv3+ for TS segmentation problems\n* State of the art results", "summary_of_the_review": "While the authors clearly put a lot of time an energy into this work, I think it would benefit from workshopping with others in the field. As noted above, there are many missing pieces from different parts of the literature which could be used to improve this work and better situate it with other progress in this area. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6elit_UwCnh", "reviewer": "Reviewer_DKC1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n-\tCompared to the sliding window approach, the proposed Segtime method can take the whole input sequence at once. This can increase the possibility of capturing long-term dependencies.\n-\tThe model combines dilated convolution and skip-LSTM for prediction, which results in better prediction results compared to the case where only a single module is used.\n\nCons\n-\tThe technical novelty of this paper is a bit limited. The proposed method is based on the 1D version of deeplabv3+ [3], with an additional MSS-LSTM module, but there are already other time-series segment approaches that are based on deep convolution layers [1, 2].\n-\tMoreover, the effect of MSS-LSTM is not thoroughly analyzed from the experiment, as SegTime has a limited effect on both datasets compared to the model without MSS-LSTM (SegTime*). Although the authors provide an ablation study (Table 4) for the effect of MSS-LSTM, the results presented in the previous experiments (Tables 2 and 3) show minor differences between SegTime and SegTime*. The authors need to provide clarification for the ablation study to prove the effect of MSS-LSTM, which is the main contribution of this paper.\n-\tThere is insufficient evidence of the effect of considering long-term dependencies. Mainly MSS-LSTM and 1D-DS-Resnet take long-term dependencies into account, but the effect on the final prediction is not properly evaluated. To consider this, the authors may report the performance according to different input lengths.\n-\tFurthermore, empirical results on the reduction of computation cost need to be provided. The authors argue that the model achieves computational efficiency by reducing parameters and computations. However, the paper does not provide appropriate experiments, such as inference time \n-\tFor clarity, I recommend the authors to correct the minor typos and grammar issues in the paper.\n\n[1] Francisco Javier Ordonez and Daniel Roggen. Deep convolutional and LSTM recurrent neural ˜ networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016.\n\n[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015.\n\n[3] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoderdecoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801–818, 2018.\n", "summary_of_the_paper": "The paper presents a stepwise segmentation for time series data, namely SegTime. Contrary to the sliding window approach, SegTime takes the whole sequence as input, and process it in two separate modules: the MSS-LSTM network and the 1D encoder-decoder network. Outputs from the two separate networks are concatenated and taken as the input to the final convolutional layer to produce the final output, which has class labels for each time-step segment. This paper’s contributions can be summarized as follows: \n(1)\tBy predicting in a step level rather than a sliding window, it works well on both fast- and slow-changing labels. \n(2)\tHigh parameter efficiency is achieved with depthwise separable convolution, atrous convolution, and skip-LSTMs.\n(3)\tLong-term dependency is captured, using MSS-LSTM and 1D convolutional layers.\n \n", "summary_of_the_review": "The SegTime method presented in the paper might be effective as the model takes the whole input sequence at once, rather than a sliding window approach. However, this paper needs further experiments and evidence to properly support the author’s claims. Moreover, technical novelty is a bit limited. Therefore, my evaluation of the paper is “marginally below the acceptance threshold”. If all the issues mentioned are fully addressed, I may reconsider my assessment of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4gdfdz-4-kO", "reviewer": "Reviewer_gLxG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several concerns regarding the manuscript:\n1) The idea of LSTM with skip connections is interesting and is borrowed from image segmentation models like DeepLabv3. However, it requires the proper adjustment of the skipping factors. How one can adjust the skipping factors? I wonder whether properly adjusting the skipping factor is equally difficult to properly adjusting the window size.\n2) In general, the whole architecture seems too complex and requires a large number of hyperparameters as illustrated in the Appendix. The description is unclear in some points (for example, do you provide the whole sequence as input to the model?) and does not present the motivation behind architectural choices (why so many modules are needed). The ablation study (Table 4) indicates minor performance deterioration when a core module is omitted. \n3) Moreover, it is not easy to justify the novelty of the method since a qualitative comparison with other methods is not presented. It seems that the proposed method is the only one that does not use the sliding window mechanism. However, if overlapping sliding windows are used differing only in one time step, a stepwise classification is achieved. \n4) Experimental evaluation can be considered as limited, since only two datasets have been considered. Note that the parameters for the Sleep-EDF dataset only are presented. Have the same parameters been used for the Opportunity dataset? To better illustrate the potential of the method, the authors could also perform experiments using synthetic time series that include both fast and slow changing labels. \n5) I strongly suggest to include the term ‘supervised’ in the title. Typically, time series segmentation is considered as an unsupervised problem where class labels are not exploited. Alternatively, the term ‘stepwise classification’ better illustrates the problem that is solved. \n", "summary_of_the_paper": "The paper presents a supervised method (called SegTime) for time series segmentation that is based on stepwise time series classification. The method avoids sliding windows (which is the typical approach), thus avoids the specification of window size and stride. It also seems to be insensitive to the label changing frequency and this constitutes a major advantage over other approaches.\nThe network architecture is based on two core modules: a novel multiscale skip LSTM (called MSS-LSTM) that employs LSTMs with skip connections and a very deep CNN (called 1D-DS-ResNet). Several other modules are also included such as 1D Depthwise Separable and Atrous Convolutional layers and the Atrous Multiscale Pooling module (AMSP).\nThe method is evaluated on two datasets, one with fast changing labels and one with slow changing labels.\n", "summary_of_the_review": "The paper presents a rather complex model for stepwise time series classification. The proposed approach relies on ideas borrowed from semantic image segmentation models like DeepLab.  There several issues to be resolved related to motivation, novelty, clarity, hyperparameter specification and experimental validation.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with segmentation of time series. The paper has received quite detailed reviews and the approach seems to have several interesting aspects (interesting architecture choice, stepwise classification approach, ability of capturing long range dependencies). However, there is a consensus that the paper would definitely benefit from a further iteration before publication in ICLR or in any other similar venue. The authors in their final response have already identified the improvement points raised by the reviewers. In addition to these, I believe it would be helpful to put the contributions better into perspective with existing literature. I think all these this would require a major rewrite and I encourage the authors to make a fresh submission in a future venue.
Decision: Reject