Con4m: Context-aware Consistency Learning Framework for Segmented Time Series Classification

Authors: junru  chen
tianyu  cao
jing  xu
jiahe  li
zhilong  chen
tao  xiao
yang  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:18:20

Abstract: Time Series Classification (TSC) encompasses two settings: classifying entire sequences or classifying segmented subsequences. The raw time series for segmented TSC usually contain Multiple classes with Varying Duration of each class (MVD). Therefore, the characteristics of MVD pose unique challenges for segmented TSC, yet have been largely overlooked by existing works. Specifically, there exists a natural temporal dependency between consecutive instances (segments) to be classified within MVD. However, mainstream TSC models rely on the assumption of independent and identically distributed (i.i.d.), focusing on independently modeling each segment. Additionally, annotators with varying expertise may provide inconsistent boundary labels, leading to unstable performance of noise-free TSC models. To address these challenges, we first formally demonstrate that valuable contextual information enhances the discriminative power of classification instances. Leveraging the contextual priors of MVD at both the data and label levels, we propose a novel consistency learning framework Con4m, which effectively utilizes contextual information more conducive to discriminating consecutive segments in segmented TSC tasks, while harmonizing inconsistent boundary labels for training. Extensive experiments across multiple datasets validate the effectiveness of Con4m in handling segmented TSC tasks on MVD. The source code is available at https://github.com/MrNobodyCali/Con4m.

Main paper: 
 introduction: 
 Time Series Classification (TSC) is one of the most challenging problems in the field of machine learning. TSC aims to assign labels to a series of temporally ordered data points. These points either form a complete sequence or are subsequences (segments) resulting from the segmentation of a long time series. Existing works [50, 20] largely focus on the assumption of independent and identically distributed (i.i.d.), in which case each sequence or segment is regarded as an independent instance to be classified, not differentiating between these two settings. In fact, for many practical applications, the raw time series before segmentation for segmented TSC tasks contain Multiple classes with † Corresponding author.38th Conference on Neural Information Processing Systems (NeurIPS 2024). Lv.3 Lv. 3 Lv.4 Lv. 4 Lv.5 Lv. 5 Lv.1 Lv. 2 Lv.2 Lv. 3 Lv.3 Lv. 4 Lv.4 Lv. 5 Lv.5(b) Illustration of annotation inconsistencies among three physicians. (c) Based on the proximity to the boundary, we divide each class sequence into 5 levels, from which an equal number of segments are sampled. A one-layer MLP is trained on the segments from each level respectively for the same number of epochs. (d) We visualize the predicted probability of the trained MLP for each level. We observe that as the segments approach the boundaries, the model finds it increasingly challenging to make correct classifications, resulting in more extreme wrong predictions. This strongly underscores the significance of handling boundary segments.Varying Duration of each class (MVD). For example, in the healthcare domain, the brain signals of epileptic patients often record over several days, encompassing multiple seizure onsets, each with varying durations and intervals. In the field of activity recognition, sensors continuously record users' behavior data, including walking, riding, and running, among other activities, each with varying durations. Therefore, the characteristics of the raw MVD lead to the uniqueness of segmented TSC tasks. Given this, our research concentrates on effectively modeling segmented TSC tasks based on MVD, presenting distinctive challenges.(1) Leveraging contextual information. In contrast to TSC tasks for complete sequences, in which classified sequences are relatively independent, there exist natural temporal dependencies between consecutive classified segments for segmented TSC. We take the seizure detection task as an example, in which given the brain signals of epileptic patients, the model should identify whether a segment includes seizure waves or not. As illustrated in Figure 1 (a), given the sustained nature of seizure onsets, the model predictions for consecutive segments should exhibit coherence, with seizure and normal predictions appearing in continuous and concentrated patterns. However, mainstream TSC models [58, 71, 61] focus on the i.i.d. assumption and model the internal context within each segment to be classified, largely overlooking the dependencies between consecutive segments.In the domain of video analysis, works in temporal action segmentation (TAS) [17] have modeled the temporal dependency between different video frames and made frame-wise predictions. However, unlike the I3D features [8] used as input in these works, time series lack a unified pretrained model for feature extraction, and the dependency between segments is more variable and ambiguous. Furthermore, TAS works focus on modeling the dependency of instances from a data perspective, without explicitly leveraging contextual label information. Therefore, how to leverage contextual information in segmented TSC tasks to make more reasonable classifications is crucial and challenging.(2) Inconsistent boundary labels. In domains with precious labels, different annotators collaboratively contribute annotations. The raw annotations of MVD typically include the start and end times for each class. However, due to inherent ambiguity and a lack of unified quantification standards, for MVD, the boundaries between states are not clearly defined, or the transitional state itself represent a mixed state. Consequently, behind inconsistent labels, there is no artificially defined true label. Therefore, our work aims to harmonize this inconsistency as much as possible to reduce the instability of model training and enhance its performance. Returning to Figure 1 (b), in the seizure detection task, owing to the natural fuzzy transition from seizure onset to completely normal, different physicians have varying experiences regarding when seizure waves terminate.Furthermore, inconsistent boundary labeling causes that boundary segments with similar patterns may have opposite labels, leading to unstable model training. To validate the detrimental impact, as shown in Figure 1 (c), we divide each class sequence in the seizure detection task into 5 levels, where higher levels indicate proximity to the boundary. We then sample an equal number of balanced binary segments for each level. Subsequently, a one-layer MLP is trained for the same number of epochs on the segments from each level respectively. Figure 1 (d) visualizes the results after training. We observe that as the level increases (closer to the boundary), the model's accuracy steadily decreases, and erroneous predictions become more extreme. The results highlight the significant detrimental impact of inconsistent boundary labels on noise-free model performance.Noisy label learning (NLL) [56] aims to learn robust models from data containing corrupted labels.While the inconsistent labels in MVD are not intentionally corrupted but rather stem from implicit discrepancies due to experiential differences, NLL remains the most relevant approach to address such discrepancies. To the best of our knowledge, Scale-teaching [47] and SREA [9] are the only NLL works specifically designed for time series and are thus the most relevant to our work. However, they also face the issue of overlooking contextual dependencies across consecutive time segments, posing the challenge of handling inconsistent boundary labels using context during training.To overcome the challenges above, we propose Con4m-a label Consistency learning framework, which leverages effective Contextual information, achieving Coherent predictions and Continuous representations for segmented TSC tasks, while harmonizing inconsistent boundary labels for training. Specifically, we first formally demonstrate that valuable contextual information enhances the discriminative power of classification instances. Based on the insights, by incorporating prior knowledge of data locality and label coherence, we guide and constrain the model to focus on contextual information more conducive to discriminating consecutive segments in segmented TSC tasks. Meanwhile, leveraging model predictions that thoroughly encompass contextual information, Con4m progressively changes the training labels in an adaptive manner to harmonize inconsistent labels across consecutive segments. This leads to a more robust model. Our contributions are summarized as follows:(1) We are the first to propose a practical consistency learning framework Con4m for the segmented TSC based on the raw MVD.(2) By comprehensively integrating prior knowledge from the data and label perspectives, we guide the model to focus on effective contextual information. Based on context-aware predictions, a progressive harmonization approach for handling inconsistent training labels is designed to yield a more robust model. (3) Extensive experiments on three public and one private MVD datasets demonstrate the superior performance of Con4m. The Con4m's ability to harmonize inconsistent labels is further verified by the label substitution experiment and case study. 
theoretical analysis: 
 In this section, we aim to formally demonstrate the benefit of contextual information for classification tasks, and to establish the existence of an upper bound for this benefit. Consequently, by introducing prior knowledge, we can guide the model to focus on valuable contextual information more conducive to improving the benefit for segmented TSC tasks.Assuming that the random variables of the instances to be classified and the corresponding labels are denoted as x t and y t . A t represents the contextual instance set introduced for x t . x At denotes the random variable for the contextual instance set. Mutual information measures the correlation between two random variables. In a classification task, a higher correlation between instances and labels indicates that the instances are more easily distinguishable by the labels. This benefits the classification task, making it more readily addressable. Therefore, from an information-theoretic perspective, we elucidate the benefit of contextual information through the following theorem. Theorem 2.1. The more the introduced contextual instance set enhance the discriminative power of the target instance, the greater the benefit for the classification task.Proof. Firstly, we establish that the introduction of contextual information does not compromise classification tasks, i.e., it does not diminish the correlation between instances and labels.EQUATIONThe inequality holds due to the non-negativity of conditional mutual information.According to (1), the increase in I(y t ; x At |x t ) determines the extent to which the introduction of contextual information can be beneficial for classification tasks. Expanding I(y t ; x At |x t ), we have: I(y t ; x At |x t ) = xt p(x t ) x AI(y t ; x At |x t ) ≤ xt p(x t ) x A t p(x At |x t )D * t ≤ xt p(x t )D * t .The convexity of the KL divergence also implies monotonicity, indicating that as A t approaches A * t , the KL divergence increases, leading to a greater information gain for the classification task.According to Theorem 2.1, valuable contextual information enhances the discriminative power of the instances. While the optimal instance set A * t is challenging to directly obtain or optimize, focusing the model on contextual instances more likely to be included in A * t is beneficial for enhancing the performance of the classification task. Furthermore, x At not only contains information at the data level but also encompasses information at the label level (which can be replaced with y At ). Therefore, we can guide the model to focus on contextual information more conducive to segmented TSC tasks by simultaneously introducing prior knowledge from both the data and label perspectives. 
the con4m method: 
 In this section, we introduce the details of Con4m. Based on the insights of Theorem 2.1, we introduce contextual prior knowledge of data locality (Sec. 3.1) and label coherence (Sec. 3.2) to guide the model to focus on contextual information more conducive to discriminating consecutive segments in segmented TSC tasks. In Sec. 3.3, inspired by the idea of noisy label learning, we propose a label harmonization framework to achieve a more robust model. Before delving into the details of Con4m, we provide the formal definition of the segmented TSC task in our work. Definition 3.1. Given a time interval comprising of T consecutive time points and labels, denoted as (X, Y ) = {(X 1 , Y 1 ), (X 2 , Y 2 ), . . . , (X T , Y T )}, a w-length sliding window with stride length r is employed for segmentation. (X, Y ) is partitioned into L time segments, represented as (x, y) = {(x i , y i ) = ({X (i-1)×r+1 , . . . , X (i-1)×r+w }, Majority({Y (i-1)×r+1 , . . . , Y (i-1)×r+w }))|i = 1, . . . , L}. The model is tasked with predicting segmented labels y i for each time segment x i .Local continuity is an inherent attribute of MVD, meaning each class should be locally continuous and only change at its actual boundary. Smoothing with a Gaussian kernel [18, 16, 66] promotes the continuity of representations of time segments in a local temporal window. This not only helps the model make similar predictions of consecutive segments within the same class but also aligns with the gradual nature of class transitions. Furthermore, for graph neural networks based on the homophily assumption, aggregating neighbor information belonging to the same class can improve the discriminative power of the target instance [49, 73] . Therefore, we introduce the Gaussian prior to guide the model to focus on contextual instances A t proximate to the target instance.Vanilla self-attention [15] with point-wise attention computations often fail to obtain continuous representations after aggregation. Therefore, we use the Gaussian kernel Φ(x, y|σ) as prior weights to aggregate neighbors to obtain smoother representations. Since the neighbors of boundary segments may belong to different classes, we allow each segment to learn its own scale parameter σ. Formally, as Figure 2 (a) shows, the two-branch Con-Attention in the l-th layer is:Q, K, V s , V g , σ =c l-1 W l Q , c l-1 W l K , c l-1 W l Vs , c l-1 W l Vg , c l-1 W l σ , S l = SoftMax QK ⊤ √ d , G l = Rescale 1 √ 2πσ i exp - |j -i| 2 2σ 2 i i,j∈{1,...,L} , z l s = S l V s , z l g = G l V g , z l = Fusion(z l s , z l g ),where L is the number of consecutive segments, d is the dimension of hidden representations, c l-1 ∈ R L×d is the output representations of the l -1-th layer, and W l * ∈ R d×d are all learnable matrices. Rescale(•) refers to row normalization by index i. To distinguish between two computational branches, we use g/G to represent the branch based on Gaussian prior, and s/S to represent the branch based on self-attention. S l and G l are the aggregation weights. We use the conventional attention mechanism [4] to adaptively fuse z l s and z l g . Finally, as illustrated in Figure 2 (a), by stacking the multi-head version of Con-Attention layers, we construct Con-Transformer, which serves as the backbone of the continuous encoder of Con4m to obtain final representations c. We employ learnable absolute positional encoding (APE) [23] for the input representations. In the segmented TSC task of MVD, consecutive time segments not only provide contextual information at the data level but also possess their own class information. As depicted in Figure 1(a) , considering the persistence of each class and the gradual nature of class transitions, the model's predictions should exhibit more coherence and concentration, rather than being interspersed. Therefore, we integrate and constrain the model's predictions from both the individual and holistic perspectives to achieve more coherent predictions.Neighbor Class Consistency Discrimination. In graphs, label propagation algorithms [30, 33] are often utilized to refine and smooth the predictions of neighbor instances, thereby enhancing their discrimination. Drawing inspiration from this, by weightedly aggregating predictions from similar time segments, the model can focus on contexts A t more likely to belong to the same class as the target segment. Although there is no explicit graph structure between time segments, we can train a discriminator to determine whether two segments belong to the same class. The model then aggregates the contextual class predictions based on the discriminator's outputs, thus making more robust predictions. As the right part of Figure 2 (b) shows, we formalize this process as follows: R = SoftMax [MLP 2 (c i ∥c j )] i,j∈{1,...,L} , p = SoftMax (MLP 1 (c)) , p = R:,:,1 p, where R ∈ R L×L×2 is the probability of whether two segments in the same time interval belong to the same class and (•∥•) denotes tensor concatenation. p represents the model's independent prediction for a segment, while p denotes the context-aware prediction that incorporates the results from neighboring segments. We then define the two training losses as ℓ 1 = CrossEntropy(p, y) and ℓ 2 = CrossEntropy( R, Ỹ ), where Ỹ = [1 yi=yj ] i,j∈{1,...,L} . Given that ℓ 1 and ℓ 2 are of the same magnitude, we equally sum them as the final loss.Prediction Behavior Constraint. Unlike graphs, there exists a holistic temporal relationship between consecutive time segments. Therefore, we should further constrain the overall predictive behavior along the time axis. For MVD, as Figure 1 (a) shows, within a suitably chosen time interval, consecutive segments almost span at most two classes. Therefore, we ensure the monotonicity of predictions across consecutive segments through hard constraints, thereby utilizing contextual label information y At to integrate and refine predictions across these segments.As shown in the middle part of Figure 2 (b), for each class in the predictions, there are only four prediction behaviors for consecutive segments, namely high confidence, low confidence, confidence decreasing, and confidence increasing. To constrain the behavior, we use function fitting to integrate p. Considering the wide applicability, we opt for the hyperbolic tangent function (i.e., Tanh) as our basis. Formally, we introduce four tunable parameters to exactly fit the monotonicity as:p = Tanh(x|a, k, b, h) = a × Tanh (k × (x + b)) + h,where parameter a constrains the range of the function's values, k controls the slope of the transition of the function, b and h adjust the symmetry center of the function, and x is the given free vector in the xcoordinate. We use the MSE loss to fit the contextual predictions p asℓ 3 = ∥Tanh(x|a, k, b, h) -p∥ 2 .It deserves to be emphasized that p in the process has no gradient and therefore does not affect the parameters of the encoder. Please see Appendix B for more fitting details.After function fitting, we obtain independent predictions p for each segment and constrained predictions p that leverage contextual label information. For the inference stage, we use the average of them as the final coherent predictions, i.e., ŷ = arg max (p + p)/2.Due to inherent ambiguity, the annotation of MVD often lacks quantitative criteria, resulting in experiential differences across individuals. Such discrepancies are detrimental to models and we propose a training framework to enable Con4m to adaptively harmonize inconsistent labels.Learning from easy to hard. We are based on the fact that although people may have differences in the fuzzy transitions between classes, they tend to reach an agreement on the most significant core part of each class. In other words, the empirical differences become more apparent when approaching the transitions. Therefore, we adopt curriculum learning techniques to help the model learn instances from the easy (core) to the hard (transition) part. Formally (see the diagram in Figure 1(b) ), for a continuous K-length class, we divide it into N l = 5 equally sized levels as follows:EQUATIONThen we sample the same number of time intervals from each level. The higher the level, the more apparent the inconsistency. Therefore, as the left part of Figure 2 (b) shows, during the training stage, Con4m learns the time intervals in order from low to high levels, with a lag gap of E g = 5 epochs.Harmonizing inconsistent labels. Inspired by the idea of noisy label learning, we gradually change the raw labels to harmonize the inconsistency. The model preferentially changes the labels of the core segments that are easier to reach a consensus, which can avoid overfitting of uncertain labels. Moreover, the model will consider both the independent and constrained predictions to robustly change inconsistent labels. Specifically, given the initial label y 0 , we update the labels y e = arg max p e for the e-th epoch, where p e is obtained as follows:p5 e = ω e • [p e-m ] m∈{0,...,4} , p5 e = ω e • [p e-m ] m∈{0,...,4} , p e = (1 -η) y 0 + η 1 - η 2 p5 e + η 2 p5 e ,where ω e = Rescale([exp((e -m)/2)] m∈{0,...,4} ) is the exponentially averaged weight vector to aggregate the predictions of the latest 5 epochs to achieve a more robust label update. pe-m and pe-m are the independent and constrained predictions in the e -m-th epoch respectively and • denotes the dot product. The dynamic weighting factor, η, is used to adjust the degree of label update. As the left part of Figure 2 (b) shows, η linearly increases from 0 to 1 with E η epochs, gradually weakening the influence of the original labels. Besides, in the initial training stage, the model tends to improve independent predictions. As the accuracy of independent predictions increases, the model assigns a greater weight to the constrained predictions. See the hyperparameter analysis for E η in Appendix C. 
experimental setup: 
 Datasets. In this work, we use three public [31, 7, 37] and one private MVD data to measure the performance of models. Specifically, the Tufts fNIRS to Mental Workload [31] data (fNIRS) contains brain activity recordings from adult humans performing controlled cognitive workload tasks. The HHAR (Heterogeneity Human Activity Recognition) dataset [7] captures sensor data from multiple smart devices to explore the impact of device heterogeneity on human activity recognition. The SleepEDF [37] data (Sleep) contains PolySomnoGraphic sleep records for subjects over a whole night. The private SEEG data records brain signals indicative of suspected pathological tissue within the brain of epileptic patients. More detailed descriptions can be found in Table 1 and Appendix D. Label disturbance. We introduce a novel disturbance method to the raw labels Y of the public datasets to simulate scenarios where labels are inconsistent. Specifically, we first look for the boundary points between different classes in a complete long MVD data. Then, we randomly determine with a 0.5 probability whether each boundary point should move forward or backward. Finally, we randomly select a new boundary point position from r% of the length of the class in the direction of the boundary movement. In this way, we can interfere with the boundaries and simulate label inconsistency. Meanwhile, a larger value of r% indicates a higher degree of label inconsistency. For SEEG dataset, inconsistent labels already exist in the raw data and we do not disturb it.Baselines. We compare Con4m with state-of-art models from various domains, including two noisy label learning (NLL) models for time series classification (TSC): SREA [9] and Scale-teaching [47] (Scale-T), three image classification models with noisy labels: SIGUA [28] , UNICON [36] and Sel-CL [43] , three supervised TSC models: MiniRocket [13] , TimesNet [64] and PatchTST [51] , and three temporal action segmentation (TAS) models: MS-TCN2 [42] , ASFormer [68] and DiffAct [45] . See more detailed descriptions of the baselines in Appendix E.Implementation details. We use cross-validation [39] to evaluate the model's generalization ability by partitioning the subjects in the data into non-overlapping subsets for training and testing. As shown in Table 1 , for fNIRS and SEEG, we divide the subjects into 4 groups and follow the 2 training-1 validation-1 testing (2-1-1) setting to conduct experiments. We divide the HHAR and Sleep datasets into 3 groups and follow the 1-1-1 experimental setting. Notice that SEEG data is derived from real clinical datasets and annotated by multiple experts, resulting in naturally inconsistent labels. We employ a voting mechanism which brings annotators together to collectively decide the boundaries to minimize discrepancies in test labels. Considering the high cost of this approach, we do not apply it to the training and validation sets. Therefore, we leave the test group aside and only change the training and validation groups to conduct cross-validation. Finally, we only report the mean values of cross-validation results in the main context. See more details and the full results in Appendix G.The average results over all cross-validation experiments are presented in Table 2 . Overall, Con4m outperforms almost all baselines across all datasets and all disturbance ratios. Results of symmetric disturbance. We also corrupt the labels with symmetric disturbance based on segmented labels y rather than raw MVD labels, which is commonly employed in the NLL works [62, 43, 32] of the image classification domain. As shown in Figure 3 (a), compared to our novel boundary disturbance, Con4m exhibits stronger robustness to symmetric disturbance. Even with the 20% disturbance ratio, Con4m treats it as a form of data augmentation, resulting in improved performance. This indicates that overcoming more challenging boundary disturbance aligns better with the nature of time series data. Since ambiguous boundaries are inherent to SEEG data and the majority voting procedure is costly, we limit this procedure to only one high-quality testing group in the label disturbance experiment.Besides, on the SEEG data, Con4m modifies approximately 10% of the training labels, which is a significant proportion. Therefore, it is necessary to further evaluate the effectiveness of our label harmonization process on SEEG data. Specifically, we train the TSC baselines based on the harmonized labels generated by Con4m and observe to what extent the performance of TSC models is improved. As shown in Figure 3 (b), PatchTST and TimesNet, employing deep learning architectures, are more susceptible to label inconsistency, so they obtain more significant performance improvement (4.11% and 7.53%). Unlike modified PatchTST that considers the contextual data information across consecutive segments, TimesNet only focuses on the independent segments, thus having a more dramatic improvement. In contrast, MiniRocket achieves only a 1.68% increase, indicating that MiniRocket is more robust with a non-deep learning-based simple random feature mapping.We introduce two types of model variations. (1) Preserve only one module. We preserve only the Con-Transformer (Con-T), Coherent Prediction (Coh-P), or Curriculum Learning (Cur-L) module separately.(2) Remove only one component. In addition to removing the above three modules, we also remove the function fitting component (-Fit) and η (E η = 0) to verify the necessity of prediction behavior constraint and progressively updating labels. As shown in Table 3 , when keeping one module, +Coh-P achieves the best performance with an averaged 2.78% decrease in F 1 score, indicating that introducing the contextual label information are most effective for MVD. The utility of each module varies across datasets. For example, for Sleep data, the Con-T contributes more to performance improvement compared to the Cur-L module, while the opposite phenomenon is observed for SEEG data. As for removing one component, even when we only remove the Tanh function fitting, the F 1 score significantly decreases 1.72% on average. On the Sleep-20% and SEEG data, the drop caused by -Fit is more significant than that caused by some other modules. Moreover, the model variation -η achieves the worst results (9.23% decrease in F 1 ).The results imply that during early training stages, the model tends to learn the consistent parts of the raw labels. Premature use of unreliable predicted labels as subsequent training supervision signals leads to model poisoning and error accumulation.True Positive False Positive False Negative True Negative Boundary We present a case study to provide a specific example that illustrates how Con4m works for MVD in Figure 4 . We show comparative visualization results for the predictions in a continuous time interval in the SEEG testing set. In SEEG data, we assign the label of normal segments as 0 and that of seizures as 1. As the figure shows, Con4m demonstrates a more coherent narrative by constraining the prediction behavior and aligning with the contextual data information. In contrast, Scale-T, Sel-CL and MiniRocket exhibit noticeably interrupted and inconsistent predictions. MS-TCN2 fails to identify normal segments. More impressively, Con4m accurately identifies the consistent boundary within the time interval spanning across two classes. We also utilize the C-score proposed by the ClaSP model [19] to assess the segmentation capability of models. The C-score reflects the ability of models to recognize segmentation boundaries by measuring the trade-off between precision (correctly identified change points) and recall (finding all true change points), ensuring the model captures meaningful transitions without over-segmenting or missing important splits. We compute the scores for each model across three sets of experiments on SEEG data and take the average. Notably, Con4m outperforms the other models significantly, while MS-TCN2, specifically designed for segmentation tasks, also achieves impressive scores. This verifies that the label consistency framework can harmonize the boundaries more effectively. Refer to Appendix H for more cases. 
conclusion and discussion: 
 In this work, we focus on the raw time series MVD for segmented time series classification (TSC) tasks, demonstrating unique challenges that are overlooked by existing mainstream TSC models. We first formally demonstrate that valuable contextual information enhances the discriminative power of classification instances. Based on the insights, we introduce contextual prior knowledge of data locality and label coherence to guide the model to focus on contextual information more conducive to discriminating consecutive segments in segmented TSC tasks. Leveraging effective contextual information, a label consistency learning framework Con4m is proposed to progressively harmonize inconsistent labels during training. Extensive experiments validate the superior performance achieved by Con4m and highlight the effectiveness of the proposed consistent label training framework. Our work still has some limitations. We have solely focused on analyzing and designing end-to-end supervised models. Further exploration of large-scale models would be challenging yet intriguing. Con4m is a combination of segmentation and classification, both of which are fully supervised. Exploring its application in unsupervised segmentation tasks is worthwhile. When faced with more diverse label behaviors, the function fitting module needs to engage in more selection and design of basis functions. Nevertheless, our work brings new insights to the TSC domain, re-emphasizing the importance of the inherent temporal dependence of time series. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: Time series classification (TSC). TSC has become a popular field in various applications with the exponential growth of available time series data in recent years. In response, researchers have proposed numerous algorithms [34] . High accuracy in TSC is achieved by classical algorithms such as Rocket and its variants [12, 13] , which use random convolution kernels with relatively low computational cost, as well as ensemble methods like HIVE-COTE [44] , which assign weights to individual classifiers.Moreover, the flourishing non-linear modeling capacity of deep models has led to an increasing prevalence of TSC algorithms based on deep learning. Various techniques are utilized in TSC: RNN-based methods [54, 14] capture temporal changes through state transitions; MLP-based methods [22, 65] encode temporal dependencies into parameters of the MLP layer; and the latest method TimesNet [64] converts one-dimensional time series into a two-dimensional space, achieving state-of-the-art performance on five mainstream tasks. Furthermore, Transformer-based models [67, 10] with attention mechanism have been widely used.The foundation of our work lies in these researches, including the selection of the backbone and experimental setup. However, mainstream TSC models [50, 20] are often designed for publicly available datasets [3, 11] based on the i.i.d. samples, disregarding the inherent contextual dependencies between classified segments in MVD. Although some time series models [55, 51] use patch-by-patch technique to include contextual information, they are partially context-aware since they only model the data dependencies within each time segment, ignoring the dependencies between consecutive segments.Noisy label learning (NLL). NLL is an important and challenging research topic in machine learning, as real-world data often rely on manual annotations prone to errors. Early works focus on statistical learning [1, 41, 5] . Researches including Sukhbaatar et al. [57] launch the era of noise-labeled representation learning.The label noise transition matrix, which represents the transition probability from clean labels to noisy labels [29] , is an essential tool. Common techniques for loss correction include forward and backward correction [53] , while masking invalid class transitions with prior knowledge is also an important method [26] . Adding an explicit or implicit regularization term in objective functions can reduce the model's sensitivity to noise, whereas re-weighting mislabeled data can reduce its impact on the objective [2, 69, 46] . Other methods involve training on small-loss instances and utilizing memorization effects. MentorNet [35] pretrains a secondary network to choose clean instances for primary network training. Co-teaching [27] and Co-teaching+ [70] , as sample selection methods, introduce two neural networks with differing learning capabilities to train simultaneously, which filter noise labels mutually. The utilization of contrastive learning has emerged as a promising approach for enhancing the robustness in the context of classification tasks of label correction methods [43, 72, 32] . These works primarily focus on handling noisy labels. And ensuring overall label consistency by modifying certain labels is crucial for MVD. To the best of our knowledge, Scale-teaching [47] and SREA [9] are the only NLL works specifically designed for time series. Scale-teaching designs a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. SREA trains a classifier and an autoencoder with a shared embedding representation, progressively self-relabeling mislabeled data samples in a self-supervised manner. However, they still face the issue of overlooking contextual dependencies across consecutive time segments.Curriculum learning (CL). Bengio et al. [6] propose CL, which imitates human learning by starting with simple samples and progressing to complicated ones. Based on this notion, CL can denoise noisy data since learners are encouraged to train on easier data and spend less time on noisy samples [24, 60] . Current mainstream approaches include Self-paced Learning [40] , where students schedule their learning, Transfer Teacher [63] , based on a predefined training scheduler; and RL Teacher [25, 48] , which incorporates student feedback into the framework. The utilization of CL proves to be particularly advantageous in situations involving changes in the training labels. Hence, this technique is utilized to enhance the harmonization process of boundary labels from MVD in a more stable manner.Temporal action segmentation (TAS). TAS is a critical task in video understanding and analysis. It involves segmenting an untrimmed video sequence into meaningful temporal segments and assigning a predefined action label to each segment [17] . The majority works on TAS typically take visual feature vectors, either hand-crafted (IDT) [59] or extracted from an off-the-shelf CNN backbone (I3D) [8] , as input for each frame. TAS uses sequential modeling to incorporate temporal dependencies and sequential context for improved TAS accuracy.While TAS shares some similarities with segmented TSC, there still exist some problems. Time series data often have high sampling rates, however, it is impossible to input excessively long sequences into TAS models designed for video data. Moreover, unlike the I3D features [8] used as input in these works, time series lack a unified pretrained model for feature extraction. Besides, the changes between adjacent video frames are smooth and continuous, whereas time series are more variable and exhibit more ambiguous boundaries. Also, TAS works focus on modeling the dependency of instances from a data perspective without explicitly leveraging contextual label information.To fit the hyperbolic tangent function (Tanh), we use the mean squared error (MSE) loss function. In practice, we use the Adam optimizer with a learning rate of 0.1 to optimize the trainable parameters. The maximum number of iterations is set to 100, and the tolerance value for stopping the fitting process based on loss change is set to 1e -6. Sequences belonging to one minibatch are parallelized to fit their respective Tanh functions. To adapt to the value range of the standard Tanh function, we rescale the sequential predictions to [-1, 1] before fitting.However, it can be difficult to achieve a good fit when fitting with the Tanh function. Specifically, random initialization may fail to fit the sequential values properly when a long time series undergoes a state transition near the boundary. For example, as Figure 5 (a) shows, we fit a sequence in which only the last value is 1. We set all default initial parameters as 1 and fit it. It can be observed that the fitting function cannot properly fit the trend and will mislabel the last point. Appropriate parameter initialization is needed to avoid excessive bias. After careful observation, we find that parameter k controls the slope at the transition part of Tanh, and parameter b controls the abscissa at the transition point. In the process, all fitting values are assigned with uniform abscissa values. Therefore, we calculate the maximum difference between adjacent values and the corresponding position in the entire sequence. And these two values are assigned to parameters k and b, respectively. This allows us to obtain suitable initial parameters and avoid getting trapped in local optima or saddle points during function fitting. Formally, given the L-length input sequence p, we initialize parameters k and b as follows:di = [p i+1 -pi ] i∈{1,...,L-1} , k, b = max (Abs(di)), arg max (Abs(di)), k = k × Sign(di[b]), b = -(b -⌊L/2⌋ + 0.5) ,where Abs(•) and Sign(•) denote the absolute value function and sign function respectively. di is the difference vector. After proper initialization, as Figure 5 (b) shows, we can obtain more accurate fitting results to reduce the probability of mislabeling. We also show some other cases (Figure 5(c)(d) )for the fitting results to verify the effectiveness of the fitting process we propose. The dynamic weighting factor η is introduced to progressively update the labels, preventing the model from overly relying on its own predicted labels too early. To validate the utility of η and determine an appropriate growth epoch E η , we conduct the hyperparameter search experiment on SEEG data. As shown in Figure 6 (b), with smaller E η (corresponding to a higher growth rate), there is a significant improvement in model performance. This aligns with our motivation that during the early stage of model training, the primary objective is to better fit the original labels. At this stage, the model's own predictions are unreliable. If the predicted results are used as training labels too early in subsequent epochs, the model would be adversely affected by its own unreliability. On the other hand, excessively large E η leads to a slower rate of label updates, making it more challenging for the model to timely harmonize inconsistent labels. Nonetheless, considering the impact of variance, the model exhibits robustness to slightly larger E η . In this work, we uniformly use E η = 30 as the default value.fNIRS. All signals are sampled at a frequency of 5.2Hz. At each time step, they record 8 real-valued measurements, with each measurement corresponding to 2 concentration changes (oxyhemoglobin and deoxyhemoglobin), 2 types of optical data (intensity and phase), and 2 spatial positions on the forehead. Each measurement unit is a micromolar concentration change per liter of tissue (for oxy-/deoxyhemoglobin). They label each part of the active experiment with one of four possible levels of n-back working memory intensity (0-back, 1-back, 2-back, or 3-back). More specifically, in an n-back task, the subject receives 40 numbers in sequence. If a number matches the number n steps back, the subject is required to respond accordingly. There are 16 rounds of tasks, with a 20-second break between each task. Following Huang et al. [31] , we only apply classification tasks for 0-back and 2-back tasks in our work. Therefore, we only extract sequences for 0-back and 2-back tasks and concatenate them in chronological order.HHAR. The HHAR (Heterogeneity Human Activity Recognition) dataset [7] captures sensor data from multiple smart devices to explore the impact of device heterogeneity on human activity recognition. It involves measurements from accelerometers and gyroscopes, which are two common motion sensors found in smartphones and smartwatches. The dataset includes six types of human activities: 'Biking', 'Sitting', 'Standing', 'Walking', 'Stairs Up', and 'Stairs Down'. For each activity, both 3-axis accelerometer and 3-axis gyroscope readings are recorded, resulting in six key features per time step. The data was collected from five different device types, consisting of four smartwatches and eight smartphones, across nine participants. Each device samples data at its highest possible frequency, generally around 50Hz. For data preprocessing, the accelerometer and gyroscope signals are aligned based on their timestamps, normalized within each channel, and divided by the devices, following the procedure outlined in Gagnon-Audet et al. [21] . To ensure balanced samples, we combined data from the Galaxy S3 Mini, LG watch, and Gear watch into a single group.Sleep. The Sleep-EDF database records PolySomnoGraphic sleep data from 197 subjects, including EEG, EOG, chin EMG, and event markers. Some data also includes respiration and temperaturerelated signals. The database contains two studies: the Sleep Cassette study and the Sleep Telemetry study. The former records approximately 40 hours of sleep from two consecutive nights, while the latter records around 18 hours of sleep. Well-trained technicians manually score the corresponding sleep graphs according to the Rechtschaffen and Kales manual. The data is labeled in intervals of 30 seconds, with each interval being marked as one of the eight possible stages: W, R, 1, 2, 3, 4, M, or ?. In our work, we utilize only the data from the Sleep Cassette study, and retain only the signals from the EEG Fpz-Cz channel and EOG horizontal channel. The EEG and EOG signals were sampled at a frequency of 100Hz. Following Kemp et al. [37] , we remove the labels for stages ? and M from the data, and merge stages 3 and 4, resulting in a 5-classification task.SEEG. The private SEEG data records brain signals of suspected pathological tissue within the brains of seizure patients. They are anonymously collected from a top hospital we cooperate with. For a patient suffering from epilepsy, 4 to 11 invasive electrodes with 52 to 153 channels are used for recording signals. In total, we have collected 847 hours of SEEG signals with a high frequency (1, 000Hz or 2, 000Hz) and a total capacity of 1.2TB. Professional neurosurgeons help us label the seizure segments for each channel. Before sampling for the database, we remove the bad channels marked by neurosurgeons. Then we uniformly downsample the data 250Hz and use a low-pass to process the data with a cutoff frequency of 30Hz. Finally, we normalize and sample the intervals for each channel respectively.• SREA [9] : This time series classification model with noisy labels jointly trains a classifier and an autoencoder with shared representations. It gradually corrects the mislabelled data samples during training in a self-supervised fashion. We use the default model architecture from the source code provided by the author (https://github.com/Castel44/SREA).• Scale-teaching [47] : This work designs a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. It uses well-learned multi-scale time series embeddings for noise label correction at sample feature levels. We modify the code to match our datasets based on the code provided by the author (https://github.com/qianlima-lab/Scale-teaching).• SIGUA [28] : This model adopts gradient descent on good data as usual, and learning-rate-reduced gradient ascent on bad data, thereby trying to reduce the effect of noisy labels. We modify the network for time series data based on the open source code provided by SREA, using the code from the author (https://github.com/bhanML/SIGUA).• UNICON [36] : UNICON introduces a Jensen-Shannon divergence-based uniform selection mechanism and uses contrastive learning to further combat the memorization of noisy labels. We modify the model for time series data according to the code provided by the author (https://github.com/nazmul-karim170/UNICON-Noisy-Label)• Sel-CL [43] : Selective-Supervised Contrastive Learning (Sel-CL) is a latest baseline model in the field of computer vision. It selects confident pairs out of noisy ones for supervised contrastive learning (Sup-CL) without knowing noise rates. We modify the code for time series data, based on the source code provided by the author (https://github.com/ShikunLi/Sel-CL)• MiniRocket [13] : Rocket [12] achieves state-of-the-art accuracy for time series classification by transforming input time series using random convolutional kernels, and using the transformed features to train a linear classifier. MiniRocket is a variant of Rocket that improves processing time, while offering essentially the same accuracy. We use the code interface from the sktime package (https://github.com/sktime/sktime).• TimesNet [64] : This model focuses on temporal variation modeling. With TimesBlock, it can discover the multi-periodicity adaptively and extract the complex temporal variations from transformed 2D tensors by a parameter-efficient inception block. We use the code from the TSlib package (https://github.com/thuml/Time-Series-Library).• PatchTST [51] : This is a self-supervised representation learning framework for multivariate time series by segmenting time series into subseries level patches, which are served as input tokens to Transformer with channel-independence. We modify the code to achieve classification for each patch, based on the source code from the Time Series Library (TSlib) package (https://github.com/thuml/Time-Series-Library).• MS-TCN2 [42] : This work proposes two multi-stage architectures for the temporal action segmentation task. While the first stage generates an initial prediction, this prediction is iteratively refined by the higher stages. Instead of the commonly used temporal pooling, they use dilated convolutions to increase the temporal receptive field. We modify the code for time series data, based on the source code provided by the author (https://github.com/sj-li/MS-TCN2).• ASFormer [68] : ASFormer is a Transformer-based model for action segmentation tasks. It explicitly brings in the inductive priors of local connectivity and applies a pre-defined hierarchical representation pattern to handle long input sequences. The decoder is also carefully designed to refine the initial prediction from the encoder. We modify the code for time series data, based on the source code provided by the author (https://github.com/ChinaYi/ASFormer).• DiffAct [45] : In this work, action predictions are iteratively generated from random noise with input video features as conditions. It also devises a unified masking strategy for the conditioning inputs to enhance the modeling of striking characteristics of human actions. We modify the code for time series data, based on the source code provided by the author (https://github.com/Finspire13/DiffAct).The non-linear encoder g enc used in Con4m is composed of three 1-D convolution layers. The number of kernels vary across different data and you can find corresponding parameters in the default config file of our source code. We construct the Con-Transformer based on the public codes implemented by HuggingFace1 . We set d=128 and the dimension of intermediate representations in FFN module as 256 for all experiments. The number of heads and dropout rate are set as 8 and 0.1 respectively.Since we observe that one-layer Con-Attention can fit the data well, we do not stack more layers to avoid overfitting. Note that Con4m consists of two Con-Transformers and we use two Con-Attention layers. The model is optimized using Adam optimizer [38] with a learning rate of 1e-3 and weight decay of 1e-4, and the batch size is set as 64. We build our model using PyTorch 2.0.0 [52] with CUDA 11.8. And the model is trained on a workstation (Ubuntu system 20.04.5) with 2 CPUs (AMD EPYC 7H12 64-Core Processor) and 8 GPUs (NVIDIA GeForce RTX 3090).The full results of the label disturbance experiment are listed in Table 4 , 5, 6 and 7. For fNIRS, we first divide the data into 4 groups by subjects and follow the 2 training-1 validation-1 testing (2-1-1) setting to conduct cross-validation experiments. Therefore, there are C 2 4 × C 1 2 = 12 experiments in total. Similarly, we divide the HHAR and Sleep data into 3 groups and follow the 1-1-1 experimental setting. Therefore, we carry out C 1 3 × C 1 2 = 6 experiments. For SEEG data, we follow the same setting as fNIRS. Notice that for SEEG data, inconsistent labels already exist in the raw data. We obtain a high-quality testing group by using a majority voting procedure to determine the boundaries. Then we leave the testing group aside and only change the validation group to report the mean value of C 2 3 = 3 experiments. All the experimental results are listed in lexicographical order according to the group name composition. We also report the mean value and standard derivation of all experiments. Specifically, we use the STDEVA to estimate standard deviation based on a sample of data. As shown in Figure 7 , we present four cases to compare and demonstrate the differences between our proposed Con4m and other baselines. The first two cases involve transitions from a seizure state of label 1 to a normal state of label 0. The third case consists of entirely normal segments, while the fourth case comprises entirely seizure segments. As illustrated in the figure, Con4m exhibits more coherent narratives by constraining the predictions to align with the contextual information of the data. Moreover, it demonstrates improved accuracy in identifying the boundaries of transition states. In contrast, other baselines exhibit fragmented and erroneous predictions along the time segments. This verifies that Con4m can achieve clearer recognition of boundaries, and it can also make better predictions on the consecutive time segments belonging to the same class. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/huggingface/transformers/blob/v4.25.1/src/transformers/models/bert/modeling_bert.py 


Decision: This paper should be accepted as a poster because it introduces a novel framework, Con4m, for time-series classification and temporal action segmentation that leverages contextual information to enhance prediction accuracy. The proposed method creatively combines time-series classification and segmentation into a unified framework, which is valuable for dealing with complex sequential data with inconsistent labels. The originality of the work lies in using a Gaussian kernel to encode contextual prior information, which is more realistic than the assumption of independent and identically distributed time series segments commonly made by other models. The authors also conducted extensive experiments on various datasets, demonstrating the framework's effectiveness in improving coherence and robustness against label disturbances. While the results are promising, they also highlight several areas for future improvement.

However, the paper does have some limitations that should be addressed. Some parts of the paper lack clarity, such as the need for clearer explanations of specific terms and the overall architecture. Moreover, there are concerns regarding the model's scalability and efficiency, especially in large-scale applications where computational and memory resources could be a constraint. The evaluation setup could also be more comprehensive, including comparisons with models that specifically address change-point detection and segmentation, which would provide a more robust assessment of Con4m's capabilities. Despite these limitations, the paper presents a sound approach with a solid theoretical foundation and holds potential for future research and application, making it a suitable candidate for a poster presentation.

Reviews: review: paper_summary: paper_summary: The authors proposed a learning framework called $\textit{Con}4\textit{n}$ that leverages contextual prior of Multiple classes with Varying Duration (MVD) to enhance the discriminative power of consecutive time series segments while harmonizing inconsistent labels associate to these later. The authors stated and shown through extensive experiment that their framework enables to encode underlying temporal dependence between consecutive time series segments and mitigate the data annotation error that may be due to discrepancy of expertise.
strength_weakness: strength_weakness: - Figure 2 is not intuitive enough to understand the overall architecture of the model;

- $\hat{p}$ and $\tilde{p}$ are mentioned without explanation. Even though for $\hat{p}$  it is quite obvious, it must be explained clearly (see paragraph $\textbf{Neighbor Class Consistency Discrimination}$.

- Space and Time complexity: Although efficient, the proposed model may require significant memory and computational time in large-scale applications.

- The model may fail when confronted with seasonal time series data. This is because the Gaussian kernel used is not suitable for capturing the periodic pattern of the data;

- Execution time is not included in the results. This is an important measure to report, as it plays a crucial role in sensitive areas such as medicine, where accuracy and prediction time are just as important.
questions: questions: - What features do you use in positional encoding? Is it the order index of time series segments or timestamps? Please mention it in the main text;

- Can the model work with multivariate time series data, i.e. using the observation of various features in a time segment to perform a classification task? If so, how will you encode the multivariate time series segments?

- Do the authors think that their model will still work correctly when faced with seasonal time series data, given that the Gaussian Kernel used is not appropriate for capturing the periodic pattern of the data?  

- Why did the authors choose the attention mechanism to merge $z_s^l$ and $z_g^l$? Although probably less efficient, have they tried simple techniques such as concatenation, Hadamard product or addition, which have the advantage of reducing spatial and temporal complexity?
limitations: limitations: Although the authors have discussed the limitations of their work, the space and time complexity of the model is an additional limitation. Furthermore, unless the reviewer has missed some points, it appears that the proposed model has only been evaluated on univariate time series data (please correct the reviewer if he is wrong). The reviewer suggests investigating these points in future work.
paper_summary: paper_summary: The paper proposes Con4m, a novel framework for time-series classification and temporal action segmentation that leverages contextual information. The framework is designed to improve the prediction accuracy by incorporating context from surrounding data segments.
The proposed method combines time-series classification (TSC) and temporal action segmentation (TAS) in a unified framework, making it versatile for different types of sequential data. It tries to address the issue of common time series classification methods which overlook the dependencies between consecutive segments.
The method demonstrates robustness to label disturbances, showing significant improvements over baseline models across different datasets (fNIRS, Sleep, and SEEG).
strength_weakness: strength_weakness: 1.	What if combining other models with a change-detection model to detect segment boundaries first and then classify the windows within the segments? That would be great to see the impact of Con4m in comparison with other baselines in this setup. 
2.	Since Con4m is capable of doing segmentation and classification together, it will be necessary to evaluate and compare the impact of each with other SOTA.
Minor change:
-	Figure 4(b) has been referred to before Figure 4(a). Please fix that
3. The experiments are limited, and somewhat unfair. The chosen baselines do not claim ability in segment detection. There are several works that have done segment or change point detection, and they need to be included in baselines. Also, evaluations are limited to three datasets.
-	Minor suggestion: It would be nice to visualize the whole framework to show each module.
questions: questions: See the initial two comments (no 1 and no 2) in weaknesses. In addition:

3.	In Label Disturbance Experiments: It is not clear how the authors evaluated the models. Is it based on the new disturbed boundaries or the original boundaries? What is the evaluation setup for the other baselines as they are not considering the labelled boundaries in their calculations (I assume they should be boundary-agnostic).
4.	While section 4.3 shows an interesting set of experiments, I believe it only measures the degree of agreement between the baseline models and Con4m about the modified labels. Unless authors can ensure all the labels modified by Con4m are correct and accurate
limitations: limitations: The following limitations are not considered/discussed:

-	Although I appreciate the novelty of the method, the final improvement is subtle compared to other baselines given they have not claimed any ability in segment detection. Also, the evaluations are limited to 3 datasets.
-	As the authors mentioned in the conclusion the model heavily relies on the availability of labels even for the segmentation part. While most segmentation models are unsupervised, I suggest including comparison by combining other baselines with unsupervised segmentation methods.
paper_summary: paper_summary: This paper addresses the challenge of segmented time series classification (TSC) for Multiple classes with Varying Duration (MVD) data. The authors propose Con4m, a consistency learning framework that leverages contextual information with a focus on inconsistent boundary labels. The method incorporates continuous contextual representation encoding, context-aware coherent class prediction, and a label consistency training framework. Experiments on 3 datasets demonstrate Con4m's superior performance compared to state-of-the-art baselines, especially in handling inconsistent labels.
strength_weakness: strength_weakness: - Limited exploration of hyperparameters: The paper doesn't thoroughly discuss the sensitivity of the method to key hyperparameters, such as N_l = 5 in the label consistency framework or E_g = 5 for curriculum learning.

- Scalability concerns: The paper doesn't address how the method would scale to larger datasets or longer time series. This could be a limitation for practical applications with high-dimensional or long-duration data.

- Comparison with semi-supervised methods: Given that the method deals with inconsistent labels, it might be beneficial to compare it with semi-supervised learning approaches that are designed to handle partially labeled or noisy data.

- Generalization to other domains: While the method is tested on three datasets, they are all from the healthcare domain (all 3 of them focusing on brain signal analysis). It would be valuable to see how well the approach generalizes to other domains with MVD data, such as activity recognition or financial time series. For example, initiatives like WOODS (https://arxiv.org/abs/2203.09978/) offer diverse timeseries data for evaluations.
questions: questions: 1. In the label consistency framework, why is N_l = 5? Was this value determined through hyperparameter search? How sensitive is the method to this choice?

2. Why was fNIRS not used in the ablation studies? Is there a specific reason for excluding this dataset from the detailed analysis?

3. How does the computational complexity of Con4m compare to the baseline methods? Is there a significant increase in training time or memory requirements?

4. Have you explored the potential of applying Con4m to semi-supervised or few-shot learning scenarios, where only a portion of the data has reliable labels?

5.  The paper mentions that Con4m modifies approximately 10% of the training labels in the SEEG data. How does this percentage vary across different datasets or disturbance ratios? Is there a way to estimate the optimal percentage of labels that should be modified?

6.  Could you provide more insights into the choice of the hyperbolic tangent function for prediction behavior constraint? Have you experimented with other monotonic functions, and if so, how do they compare?
limitations: limitations: See the questions above. I would especially emphasize the narrow benchmarks and tasks. Learning across segments is a broader problem within the timeseries domain.
paper_summary: paper_summary: This paper proposes a time-series classification method which exploits temporal consistency (implemented by contextual information). Also, the proposed method can handle noisy class boundaries.
strength_weakness: strength_weakness: W1. First of all, the writing in Introduction and Figure 1 are very misleading. "**Inconsistent** boundary labels" (with "different annotators") and Figure 1(a) indicate that multiple inconsistent annotations coexist for the same time-series instance. However, after reading a few pages, I found a discrepancy between my understanding and the presentation in this paper. Subsequently, I realized that the authors were referring to scenarios in which the class boundaries could be inaccurate. That is, there is **one** possibly erroneous boundary label for each transition. This presentation issue really affected my negative impression on this paper.

W2. I do not agree that existing studies largely focus on the assumption of i.i.d. It is very easy to find the existing work that considers temporal consistency. For example, please see https://cs.stanford.edu/people/jure/pubs/ticc-kdd17.pdf. More recently, please see https://openreview.net/pdf?id=gjNcH0hj0LM. That is, many existing methods consider that consecutive timestamps tend to have the same class label. 

W3. Con-Attention is the same as Anomaly-Attention. Please see Equation (2) in https://arxiv.org/pdf/2110.02642. Therefore, along with W2, the technical novelty for the first contribution is not very significant.

W4. Section 2 is not tightly connected to subsequent sections. In my opinion, Theorem 2 is too obvious, and it does not need to be formulated as a theorem. In fact, it does not look like a formal theorem. 

W5. The Tanh function fitting seems to be needed for each segment boundary and is not straightforward (involved with many parameters). Thus, the training efficiency could be an issue. It would be good to report the training cost of the proposed method.

W6. The authors assume that the center of a class interval has the highest confidence. This assumption could be true or false depending on the domain. As in the motivating example, if the annotators have a difficulty in identifying the end times of seizure waves, the highest confidence will appear earlier than the center.

W7. The authors made several strong assumptions or heuristics. For example, it is assumed that consecutive segments almost span at most two classes; a class interval is divided into five levels; a training cap is given as five epochs. Thus, this paper lacks rigorousness and generalizability. 

W8. It is not trivial to set up the hyperparameter value, especially, $E_\eta$. The default value is not guaranteed to work well for other datasets. A detailed guideline is necessary.

W9. The degree of noisy labels in each dataset should be analyzed in detail. First, it is not clear why there is no noise in fNIRS and Sleep, while there is real noise in SEEG. Is the annotation strategy different in these datasets? How do you know the ground-truth labels in SEEG? Second, can you measure the degree of real noise just like the noise rate in the image classification domain? Can you approximate the corresponding value of $r$ for this degree of noisy labels in SEEG? Overall, without detailed information on noisy labels, it would be difficult to analyze the effect of noisy label learning.

---

I have adjusted my rating to 5 during the discussion period.
questions: questions: See W1~W9.
limitations: limitations: In my opinion, the generalizability of the proposed method is not clearly verified. In this regard, the authors did not properly discuss the limitations of the proposed method. Please see my comments to further improve the manuscript.

Comments: title: Thank you
Further discussion
Thank you
Thanks for your clarification
Further discussion
Thank you
Thank you
Satisfied with the answers.
Results for unsupervised segmentation model ClaSP
Will keep my current rating
comment: Thank you for your response and score improvement.

As ClaSP is only capable of handling univariate time series data, we conduct experiments solely on the SEEG dataset, as the other three publicly available datasets consist of multivariate time series. We are still perplexed as to why Con4m should be compared with unsupervised segmentation models. From our perspective, Con4m must rely on independent sample predictions $\hat{p}$ and contextual predictions $\tilde{p}$ to perform function fitting to determine the class boundary. However, unsupervised segmentation models cannot specify the exact classes of the samples, while the Temporal Action Segmentation (TAS) models can achieve this. Therefore, we provide an exposition (lines 43-49) and comparison of the TAS models. Nevertheless, we concur that this comparative result can be utilized for the quantitative metrics in Section 4.5 to highlight Con4m's performance in segmentation precision and coherence. Consequently, we will include ClaSP's F1 scores in Figure 5. Additionally, at line 334, we will insert: "Con4m is a combination of segmentation and classification, both of which are fully supervised. Exploring its application in unsupervised segmentation tasks is worthwhile." Regarding the experimental setup, we will supplement it in accordance with the point (3) in the response (https://openreview.net/forum?id=jCPufQaHvb&noteId=fNo5b5RBTi) with Reviewer Gi9s.
The authors have added a comparison with an unsupervised segmentation method. However, the comparison is only done across one of the datasets out of three.  
Overall, the authors' approach is a combination of segmentation and classification, both are fully supervised. 
These limitations need to be addressed and clarified in the paper, as there are existing approaches for change point detection that are self-supervised. 

I also agree with the last reviewer's comment. The authors need to improve the work by explaining the experiment setup more clearly.

Given the edits and discussions, I am happy to increase the score to 5.
We would like to reiterate the connection and distinction between unsupervised segmentation tasks and our scenario:

(1) Pure segmentation models can only identify the positions of change points, without specifying the exact classes on either side of the change points, especially in multi-class scenarios. Although Con4m was not specifically designed for segmentation tasks, it can still transform from segment prediction to point prediction and provide segmentation results. Therefore, based on the SEEG dataset, we evaluate Con4m fairly according to the F1 scores provided by ClaSP. Experimental results are as follows: ClaSP: 0.854; Con4m: 0.930. We believe this result corroborates the conclusion drawn in Section 4.5.

(2) Temporal Action Segmentation (TAS) models are specifically designed for video segmentation. Following your indication of being "capable of doing segmentation and classification together," the TAS models perfectly fulfill both criteria as they can provide a specific class prediction for each video frame/time segment. Hence, our choice of baseline models is comprehensive and fair.

(3) Many segmentation models are primarily designed for univariate time series and face challenges when handling multivariate time series data. Therefore, we could only compare ClaSP on the unique univariate dataset SEEG.

As we are not experts in the field of unsupervised segmentation, we welcome any corrections if there are any inaccuracies or omissions. Additionally, we would greatly appreciate your timely communication and sharing of thoughts that integrate such models into our scenario.
Thank you for your trust and raising the score.

(1) Regarding the problem setup, we will emphasize the meaning of inconsistent labels starting from line 52: "However, due to inherent ambiguity and a lack of unified quantification standards, for MVD, the boundaries between states are not clearly defined, or the transitional states themselves represent a mixed state. Consequently, behind inconsistent labels, there is no artificially defined true label. Therefore, our work aims to harmonize this inconsistency as much as possible to reduce the instability of model training and enhance its performance."

(2) Regarding Figure 1, our current plan for modification is as follows: "Place (c) at the top and elongate it, then vertically superimpose two other distinct but similar types of seizure waves. The annotations from three physicians will be marked on these three brain signals. The labels lv.1-lv.5 will be placed inside the signals to reduce visual clutter. The three signals collectively point to the x-axis of figure (d)."

(3) Regarding the experimental setup, we will insert the content about SEEG from Appendix G starting from line 254: "Notice that SEEG data is derived from real clinical datasets and annotated by multiple experts, resulting in naturally inconsistent labels. We employ a voting mechanism which brings annotators together to collectively decide the boundaries to minimize discrepancies in test labels. Considering the high cost of this approach, we do not apply it to the training and validation sets. Therefore, we leave the test group aside and only change the training and validation groups to conduct cross-validation."
I now have a better understanding on the problem setting and experiment configuration. The overall presentation should be significantly improved to avoid any confusion on the problem setting and experiment configuration. In particular, Figure 1 must be revised to properly explain the motivation. I will adjust my rating to 5, assuming that the authors will reflect my comments. Thank you very much.
Thank you for your response. We have also included the analysis of unsupervised segmentation models in our setup, along with the configuration and comparative results of ClaSP on SEEG data. Do you have any further questions or concerns regarding these responses? We are more than willing to engage in further discussion to enhance the quality and score of our work.
Thank you very much for raising your score, which is highly valuable to us. We have gained a lot from your comments, such as identifying the HHAR dataset from the WOODS benchmark as one of our test datasets. If you have any further concerns or suggestions, please do not hesitate to share them with us.
Thank you once again for your recognition and affirmation of our work. If you have any further questions or concerns, please feel free to let us know at any time.
Thank you for your response. We would like to provide further clarification.

(1) This involves defining what constitutes "erroneous." Erroneous labels imply that we assume there exists an objectively true label behind each label. However, inconsistent labels simply represent differences in annotator experience. As discussed in lines 50-56 of the main text, for MVD, the boundaries between states are not clearly defined, or the transitional states themselves represent a mixed state. Consequently, behind inconsistent labels, there is no artificially defined true label. Perhaps a model could assist in defining this true label, but thus far, our work aims to harmonize this inconsistency as much as possible to reduce the instability of model training and enhance its performance.

(2) The two works you mentioned in W2, one designed for clustering tasks and the other for active learning, both fall outside the scope of TSC. Our design for representing continuity and label coherence is two sides of the same coin, closely aligned with our intent and motivation. We conduct hyperparameter analysis for W7 in G2, and in Section 5, we spell out the limitation regarding the diversity of class transition behaviors.

(3) As mentioned in (1), there is no objectively quantifiable true labels behind inconsistent labels, making it challenging to quantify inconsistency. The voting mechanism refers to bringing multiple experts together to collectively decide the boundaries of the test set, rather than independently annotating different patients or files. However, this approach incurs significant costs, as each patient's record is extensive (spanning several days or even a dozen days), hence we can only apply it to the test set. If the training set were also handled in this manner, we have reason to believe that SEEG data could be considered equivalent to clean datasets like fNIRS and Sleep.
I would like to thank the authors for responding to my questions. The provided clarifications for Q3 and Q4 makes the evaluation setup more clear.
I want to thank the authors for their thorough responses. I have updated my score accordingly.
Dear authors, thank you for your detailed responses. My concerns were clarified.
Apologies for the delayed results. For the F1 score provided by ClaSP, we input each time interval into ClaSP to obtain a score, and then average the scores of all time intervals. Additionally, we transform Con4m's predicted results from time segments to timestamps and use them to calculate the F1 score proposed by ClaSP. The specific scores are as follows: ClaSP: 0.854; Con4m: 0.930. The comparison of these results is quite fair.

To align with our evaluation metrics, we assign all possible class combinations to the timestamps on either side of the change points detected by ClaSP. We then select the combination with the greatest overlap with the true test labels for ClaSP. Finally, the same segmentation and evaluation processes are executed to obtain the results. The specific f1 scores are as follows: ClaSP: 0.824; Con4m: 0.720. While this approach provides a test result, it clearly involves test label leakage, rendering it unfair and unsuitable for real-world applications.
Thank you for your responses. I have carefully read the authors' rebuttal.

(1) As I mentioned earlier, the overall presentation is somewhat confusing. I now better understand the authors' original intention. However, the current draft needs a major rewrite. The differences between erroneous and inconsistent boundary labels are still not clear to me. To my understanding, this paper deals with possibly erroneous boundary labels caused by inconsistent annotations. Because we cannot review the revised draft, it would be hard to increase the current rating.

(2) The responses on the novelty issues (W2 and W3) and the generalizability issues (W7) are not very convincing to me.

(3) The in-depth analysis w.r.t. the degree of noise or inconsistency (W9) would be necessary to further strengthen this work. A voting mechanism applied to SEEG means curation? If you apply the voting mechanism to the training set of SEEG, it becomes the same status as fNIRS and Sleep? There are several unclear points regarding the motivation and experiment setting. 

Overall, I would like to keep my current rating.