Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new deep architecture for segmentation of time series, i.e. to find the subsequences inside of a time series corresponding to different classes and to determine the bounds of those subsequences. The proposed architecture includes two core components: a) a kind of LSTM with skip connections to deal with the multi-scale problem of time series, b) an encoder-decoder module based on CNN and ResNet. The outputs of the two components are then used in a convolutional layer to provide a stepwise classification at each time step. The approach is evaluated on two classical datasets wrt 3 baselines.
main_review: Pros: 
* The paper addresses an important problem of time series, i.e. accurate segmentation of a time series 
* The philosophy of the presented approach (stepwise classification) seems promising
* Using both LSTM and CNN to treat the multi-scale challenge of time series is interesting, especially how it is done in the paper with two separate modules.
* The SOTA section is accurate and easy to read. 
* Having a single architecture able to deal with fast and low changing labels is clearly an important outcome of the paper.  

Neutral:
* While the architecture is novel (to the best of my knowledge), most of the used modules are adapted from existing literature.
 
Cons:
* The technical section is very hard to read, the appendixes are not used at the best to explain deeper the architecture. The main problem is that there is no explanation of the role/the idea behind each module. What can achieve the MSS-LSTM that can not achieve the encoder-decoder ? Why use AMSP, 1D-DS-ResNet ... ? Looking especially at the ablation study, I wonder if the two modules are really useful and I can not judge because the evaluation is not sufficiently detailed (see below). 
* The experimental part is insufficient to judge the relevance of the proposed architecture:
   * evaluating more datasets will be useful to assess the stability of the proposed approach wrt the SOTA. As it is, the results are very close and it is hard to assess the strength and weakness of the proposed approach wrt SOTA.
   * the core thesis of the authors is to provide a model able to do stepwise segmentation and to retrieve accurately the bounds of each subsequence. Unfortunately, all results concern aggregated measures over the whole sequences like accuracy or F-score. It seems very important to have some results on the bounds found by the model compared to SOTA (maybe something like a delta-time between the ground truth and the inferred timestep ?). 
   * the datasets are not well detailed (for instance what is the proportion of each class ? the paper said that the problem is unbalanced but no further details). 
   * the results are not well detailed: for the ablation study, only one single aggregated result is reported, it is impossible to understand the real role of each component. 
   * there is no indication about the training time and the inference time of each model, and as the results are very close, it is important to know the computational cost of each model. 

summary_of_the_review: The approach seems interesting and promising but:
* the technical part is too hard to follow and requires more explanation on the reason for the use of each module 
* the experimental part is incomplete missing at least an important experiment to show how well the proposed model answer to the problem of the accurate detection of the time series segments; and details on results to understand the ablation study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a stepwise segmentation for time series data, namely SegTime. Contrary to the sliding window approach, SegTime takes the whole sequence as input, and process it in two separate modules: the MSS-LSTM network and the 1D encoder-decoder network. Outputs from the two separate networks are concatenated and taken as the input to the final convolutional layer to produce the final output, which has class labels for each time-step segment. This paper’s contributions can be summarized as follows: 
(1)	By predicting in a step level rather than a sliding window, it works well on both fast- and slow-changing labels. 
(2)	High parameter efficiency is achieved with depthwise separable convolution, atrous convolution, and skip-LSTMs.
(3)	Long-term dependency is captured, using MSS-LSTM and 1D convolutional layers.
 

main_review: Pros
-	Compared to the sliding window approach, the proposed Segtime method can take the whole input sequence at once. This can increase the possibility of capturing long-term dependencies.
-	The model combines dilated convolution and skip-LSTM for prediction, which results in better prediction results compared to the case where only a single module is used.

Cons
-	The technical novelty of this paper is a bit limited. The proposed method is based on the 1D version of deeplabv3+ [3], with an additional MSS-LSTM module, but there are already other time-series segment approaches that are based on deep convolution layers [1, 2].
-	Moreover, the effect of MSS-LSTM is not thoroughly analyzed from the experiment, as SegTime has a limited effect on both datasets compared to the model without MSS-LSTM (SegTime*). Although the authors provide an ablation study (Table 4) for the effect of MSS-LSTM, the results presented in the previous experiments (Tables 2 and 3) show minor differences between SegTime and SegTime*. The authors need to provide clarification for the ablation study to prove the effect of MSS-LSTM, which is the main contribution of this paper.
-	There is insufficient evidence of the effect of considering long-term dependencies. Mainly MSS-LSTM and 1D-DS-Resnet take long-term dependencies into account, but the effect on the final prediction is not properly evaluated. To consider this, the authors may report the performance according to different input lengths.
-	Furthermore, empirical results on the reduction of computation cost need to be provided. The authors argue that the model achieves computational efficiency by reducing parameters and computations. However, the paper does not provide appropriate experiments, such as inference time 
-	For clarity, I recommend the authors to correct the minor typos and grammar issues in the paper.

[1] Francisco Javier Ordonez and Daniel Roggen. Deep convolutional and LSTM recurrent neural ˜ networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016.

[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015.

[3] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoderdecoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801–818, 2018.

summary_of_the_review: The SegTime method presented in the paper might be effective as the model takes the whole input sequence at once, rather than a sliding window approach. However, this paper needs further experiments and evidence to properly support the author’s claims. Moreover, technical novelty is a bit limited. Therefore, my evaluation of the paper is “marginally below the acceptance threshold”. If all the issues mentioned are fully addressed, I may reconsider my assessment of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on time-series (TS) segmentation. They claim that in the typical approach for this problem -- where you is to apply a model (e.g., a temporal conv net) over fixed windows in time in sliding window fashion -- it is challenging to predict precise breakpoints, especially when the labels change frequently relative to the sampling rate of the input data/sensors. They also claim that these approaches ignore long-term dependences. They introduce a network which they "obviates" the need for sliding windows and can precisely find breakpoints.

Their claimed contributions are: 
* A conceptual framework for TS segmentation
* An architecture for solving TS segmentation problems
* An adaptation of DeepLabv3+ for TS segmentation problems
* State of the art results
main_review: This paper brings up many interesting questions and does interesting analyses. However, I have concerns primarily from the broad conceptual claims and missing background from literature in this area. 

A core conceptual insight in the paper is described in contribution #1. 
* There is a claim that most TS segmentation methods operate as "divide-classify-concatenate," however, this isn't true. Most current papers in this area use Temporal Conversion Net architectures which (as the paper also mentions) typically operate on a sliding window approach. Much of this work until a couple years ago was using LSTMs, Conditional Random Fields, HMMs, or other temporal models which also did not operate under divide-classify-concatenate. The conceptual framework used by this paper is no different than the union of TCN an LSTM-based approaches (which has also been widely documented in the literature, e.g., in the Speech community).
* There is also a claim that other works haven't looked at labels operating over multiple time-scales. This is reiterated on page 3: "No prior work has investigated time series segmentation in two scales: labels of high frequency and low frequency change. Few works are dedicated a precise segmentation with step-level accuracy." These statements highlight that this paper is missing a large number of references from the computer vision and ML communities, often under the guise of fine-grained action recognition, working to overcome many of these same problems. The MS-TCN work below, for example, specifically addresses multi-scale issues like this. 

Some good references. I recommend doing a search for more in this direction. 
* Lea et al. "Temporal Convolutional Networks for Action Segmentation and Detection" CVPR 2017.
* Bai et al. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling" arxiv 2018.
* Farha et al. "Ms-tcn: Multi-stage temporal convolutional network for action segmentation" CVPR 2019 and "MS-TCN++" at TPAMI 2020.
* Kahatapitiya et al. "Coarse-Fine Networks for Temporal Activity Detection in Videos" CVPR 2021.

Re: the stepwise segmentation module. On one hand I understand why you would want to reduce the temporal resolution of your output space (e.g., from input sampling rate of 3k hz to output of 30 hz). This is commonly done in areas like speech where acoustic models are often downsampled to word-fragments at 30 (or 100) hz. However, this seems antithetical to the premise of the paper, where the goal is to have very precise timestamps. 

Re: SegTime-. While I like the simplify of the multi-scale Skip LSTM idea, I appreciate the ablations between the full (TCN+LSTM) model and the adaptation of DeepLabv3 (TCN). While the LSTM-based model does improve performance, I wonder if making other modifications to the TCN-side of the model could have the same impact. Ablations on other TCN architectures could be useful here. It's unclear what issues the LSTM-side prevents which couldn't alternatively be done with a modified TCN architecture. 

summary_of_the_review: While the authors clearly put a lot of time an energy into this work, I think it would benefit from workshopping with others in the field. As noted above, there are many missing pieces from different parts of the literature which could be used to improve this work and better situate it with other progress in this area.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a supervised method (called SegTime) for time series segmentation that is based on stepwise time series classification. The method avoids sliding windows (which is the typical approach), thus avoids the specification of window size and stride. It also seems to be insensitive to the label changing frequency and this constitutes a major advantage over other approaches.
The network architecture is based on two core modules: a novel multiscale skip LSTM (called MSS-LSTM) that employs LSTMs with skip connections and a very deep CNN (called 1D-DS-ResNet). Several other modules are also included such as 1D Depthwise Separable and Atrous Convolutional layers and the Atrous Multiscale Pooling module (AMSP).
The method is evaluated on two datasets, one with fast changing labels and one with slow changing labels.

main_review: There are several concerns regarding the manuscript:
1) The idea of LSTM with skip connections is interesting and is borrowed from image segmentation models like DeepLabv3. However, it requires the proper adjustment of the skipping factors. How one can adjust the skipping factors? I wonder whether properly adjusting the skipping factor is equally difficult to properly adjusting the window size.
2) In general, the whole architecture seems too complex and requires a large number of hyperparameters as illustrated in the Appendix. The description is unclear in some points (for example, do you provide the whole sequence as input to the model?) and does not present the motivation behind architectural choices (why so many modules are needed). The ablation study (Table 4) indicates minor performance deterioration when a core module is omitted. 
3) Moreover, it is not easy to justify the novelty of the method since a qualitative comparison with other methods is not presented. It seems that the proposed method is the only one that does not use the sliding window mechanism. However, if overlapping sliding windows are used differing only in one time step, a stepwise classification is achieved. 
4) Experimental evaluation can be considered as limited, since only two datasets have been considered. Note that the parameters for the Sleep-EDF dataset only are presented. Have the same parameters been used for the Opportunity dataset? To better illustrate the potential of the method, the authors could also perform experiments using synthetic time series that include both fast and slow changing labels. 
5) I strongly suggest to include the term ‘supervised’ in the title. Typically, time series segmentation is considered as an unsupervised problem where class labels are not exploited. Alternatively, the term ‘stepwise classification’ better illustrates the problem that is solved. 

summary_of_the_review: The paper presents a rather complex model for stepwise time series classification. The proposed approach relies on ideas borrowed from semantic image segmentation models like DeepLab.  There several issues to be resolved related to motivation, novelty, clarity, hyperparameter specification and experimental validation.
review_title: null

Meta-review: This paper deals with segmentation of time series. The paper has received quite detailed reviews and the approach seems to have several interesting aspects (interesting architecture choice, stepwise classification approach, ability of capturing long range dependencies). However, there is a consensus that the paper would definitely benefit from a further iteration before publication in ICLR or in any other similar venue. The authors in their final response have already identified the improvement points raised by the reviewers. In addition to these, I believe it would be helpful to put the contributions better into perspective with existing literature. I think all these this would require a major rewrite and I encourage the authors to make a fresh submission in a future venue.
Decision: Reject