Toward Interpretable Evaluation Measures for Time Series Segmentation
Authors: Félix Chavelli
Paul Boniol
Michaël Thomazo
Venue: NeurIPS 2025 poster ()
Keywords: time series
state detection
change point detection
evaluation measure
interpretability
Primary area: applications

Abstract: Time series segmentation is a fundamental task in analyzing temporal data across various domains, from human activity recognition to energy monitoring. While numerous state-of-the-art methods have been developed to tackle this problem, the evaluation of their performance remains critically limited. Existing measures predominantly focus on change point accuracy or rely on point-based metrics such as Adjusted Rand Index (ARI), which fail to capture the quality of the detected segments, ignore the nature of errors, and offer limited interpretability. In this paper, we address these shortcomings by introducing two novel evaluation measures: WARI (Weighted Adjusted Rand Index), a temporal extension of ARI that accounts for the position of segmentation errors, and SMS (State Matching Score), a fine-grained metric that identifies and scores four distinct and fundamental types of segmentation errors while allowing error-specific weighting. We empirically validate WARI and SMS on synthetic and real-world benchmarks, showing that they not only provide a more accurate assessment of segmentation quality but also uncover insights, such as error provenance and type, that are inaccessible with traditional measures.

Toward Interpretable Evaluation Measures for Time Series Segmentation
Félix Chavelli
Paul Boniol Inria, ENS, CNRS, PSL Paris, France paul. boniooliaal@iii..f.
Michaël Thomazo Inria, ENS, CNRS, PSL Paris, France mi michael.o thomazo@inria.irr
Inria, ENS, CNRS, PSL
Paris, France
felix. caw.echaaelllie ieiiiiiiiia.i....
Abstract
Time series segmentation is fundamental task in analyzing temporal data across various domains, from human activity recognition to energy monitoring. While numerous state-of-ther of-tte--rttttttt methods have been developed tackle this problem, the evaluation their performance remains critically limited. Existing measures predominantly focus change point accuracy or rely on point-based measures such as Adjusted Rand Index (ARI), which fail to capture the quality of the detected segments, ignore the nature of errors, and offer limited interpretability. In this paper, we address these shortcomings by introducing two novel evaluation measures: WARI (Weighted Adjusted Rand Index), that accounts for the position of segmentation errors, and SMS (State Matching Score), a fine-grained measure that identifies and scores four fundamental types of segmentation errors while allowing error-specific weighting. We empirically validate WARI and SMS on synthetic and real-world benchmarks, showing that they not only provide more accurate assessment of segmentation quality but also uncover insights, such as error provenance and type, that are inaccessible with traditional measures.
Introduction
Massive collections of time-varying measurements, commonly referred to as time series, have become reality in every scientific and industrial domain [1-4]. Such temporal measurements can correspond different physical quantities, such as temperature and pressure [5], electricity consumption [6], or human pose [7]. Several tasks have emerged from the pressing need to analyze time series, such as classification [8], clustering [9], anomaly detection [10-12], motif discovery [13], and time series segmentation [14]. The latter (sometimes referred to as change point delection state delection) is a crucial task. The goal is to identify distinct states or patterns within the data, which can provide valuable insights into the underlying processes. More generally, time series segmentation aims to respectively detect change points, which delincate different states, and to cluster these states in order recognize recurring states. Such states correspond to human activity such as walking and rumning [15], or specific appliances in electrical consumption time series [6]. Although wide variety of oftte-of-theaare algorithms have been proposed, leveraging diverse approaches statistical methods [14, 16, 17], Markov models [18, 19), auto-encoders [20-22], or symbolic representations [23]), enabling notable progress in recent benchmarks, observe three major limitations that undermine their ability to reliably assess segmentation quality.
First, change point-based measures, which focus solely the accuracy of detecting transition points, do not adequately capture the overall quality of the segmentation itself. Even if change points correctly detected, the resulting segment labels might still be incorrect uninformative. Second, most widely used measures, such as Adjusted Rand Index (ARI), are point-based and thus treat all errors (i.e., points belonging to wrongly segmented subsequences) equally, failing to distinguish
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

between different types of errors. For instance, delay in detecting transition may simply reflect minor misalignment with human annotation, whereas an isolated error, such as labeling an entire segment incorrectly, is far more severe. These two types of errors carry very different implications, yet traditional measures weight them equally. Lastly, current evaluation measures cannot track and categorize the nature of errors (e.g., delay vs. isolation), leading to limited interpretability. This hinders deeper diagnostic and reduces the practical value of the measure for improving models.
To address these limitations, we introduce two new evaluation measures: WARI (Weighted Adjusted Rand Index) and SMS (State Matching Score). The first measure, WARI, extends the traditional Adjusted Rand Index by incorporating the temporal position of segmentation errors. This allows differentiate between positions of errors, such as between errors close but misaligned with the ground truth, and isolated eπrors which indicate more substantial segmentation failures. While WART provides a more nuanced and temporally aware variant of ARI, SMS offers a complementary perspective by explicitly identifying and scoring four fundamental types of segmentation errors. More specifically, SMS allows practitioners to weight each error type based on their application, thus providing customizable and interpretable evaluation measure. By maintaining error provenance and enabling targeted analysis, SMS enhances the transparency segmentation assessments. Finally, we empirically evaluate the validity of WARI and SMS by comparing them to existing measures. We then report the impact our measures on the evaluation of -fee-o--theerrr segmentation methods. We also provide additional insights, such the prevalence and severity of specific error types, that were previously inaccessible. Overall, our contributions are as follows:
We provide a thorough analysis of the literature on time series segmentation and propose a typology of fundamental and distinct segmentation errors (ef. Sec. 2.1 to 2.2).
We critically examine the limitations of existing evaluation measures, highlighting their inability to capture key aspects of segmentation quality (cf. Sec. 2.3).
We introduce our two novel evaluation measures, WARI and SMS, and provide detailed descriptions of their design, objectives, and theoretical advantages (ef. Sec. 3).
We empirically demonstrate that our proposed measures offer more appropriate and insightful evaluation of segmentation quality compared to existing measures (cf. Sec. 4.1).
We analyze WARI and SMS impact on the assessment of of-te-of-thear segmentation methods, and present novel insights that are impossible with traditional evaluation measures (cf. Sec. 4.2).
2 Background and Foundations
This section provides the foundational concepts and formal definitions necessary for understanding time series segmentation and evaluation strategics. We first introduce fundamental definitions to assess the technical differences between problem formulations and existing algorithms.
Definition (Real-valued Time Series) A real-valued time series of length N and dimension D is a time-ordered sequence denoted by T = "i,.,,,,N,,, where each t; E RD for = I....,N..
We define a univariate time series as a time series with D 1. Moreover, a subsequence of T from index i to (with 1 < < j < N) is denoted by T(.j] = [ts, fat1....,. and has length j -i+1 We now define a state sequence as follows:
Definition 2 (State Sequence) A state sequence S= [s, associated with a time series
T= T= .......,,yy .,]] is a sequence of the same length, where each 8; E S is a discrete label representing the latent state of the system time step and S is finite set of possible states. In state sequence, i (with < <, N) is a change point if s, + 81+1.
2.1 Time Series Segmentation: Multifaceted Problem
Time series segmentation refers to the task of dividing time series into meaningful and homogeneous segments, where each segment corresponds to a period during which the underlying generative process is assumed to be stable. Two common approaches to segmentation are change point detection and state detection, illustrated in Fig. While they differ assumptions and goals, both aim to capture shifts in the behavior in the time series.

---

Change Point Delection
CP
CP
CP
Time
State A
State B
State C
State B
State A
State Detection
Figure 1: Illustration of Change Point Detection vs. State Detection.
2.1.1
Change Point Detection
The output of a change point detection algorithm is an increasing sequence of integers (CI, ........ CM where each change point (CP) marks a transition between two segments. The resulting segments are contiguous and non-overlapping, each corresponding to a stationary or stable regime. Change point detection is typically used when the goal is to precisely localize transitions.
large panel of approaches tackling change point detection has been introduced in the literature. First, rofile-based methods, such as ClaSP [14], FLUSS [16], and ESPRESSO [17], typically operate by constructing profile from the time series and identifying CPs local extrema. Second, other methods proposed in the literature rely statistical principles. For instance, binary segmentation [24] (BinSeg) employs recursive likelihood hypothesis testing, and PELT [25] offers a pruned, optimization-aas variant. Finally, Bayesian approaches, such as BOCD [26], sequentially update the probability of CP presence as new data arrives.
2.1.2 State Detection
In contrast, state detection assumes that the time series is generated by an underlying sequence of latent states. Each state corresponds specific pattern regime, and the same state recur different points in time. Hence, the output of a state detection algorithm a predicted state sequence P P=(pa.---. (pi,...-PM) ,N... ). The primary goal of state detection is identify changes in the latent state itself, allowing for the recognition of recurring patterns, rather than simply detecting any statistical shift.
Diverse approaches have been proposed for state detection. These include methods based on encoder architectures (e.g., E2USD [20]. Time2State [21], HVGH [22]), convolutional neural networks such as RP-mask [27] and PrecTime [28], graph representations such as GRAB [29] and uGLAD [30]), probabilistic graphical models such as hidden Markov models (e.g.. HDP-HSMM [18] and MASA [19]), Markov random fields (e.g., TICC [31]), male-based systems (e.g.. PaTSS [23]).
Importantly, change point detection can be viewed subproblem of state detection. Once change points have been identified, they partition the time series into segments. A clustering algorithm can then be applied to these segments to assign state labels (as performed in [20, 21] for the change point detection method ClaSP [14]). This two-step approach enables the reconstruction of state sequence from raw change points, highlighting that state detection generalizes change point detection. Thus, for the rest of the paper, we will mainly focus the state detection problem.
2.2 Evaluating Time Series Segmentation: Typology of Errors and Desired Properties
In order to accurately assess the quality of segmentation when compared with a ground-truth, we need to formally define segmentation error types. Let S = {81,...- .M33 be a finite set of states. Let Rt = (r1, ... FN) E S be the real, state sequence, and let P p= (P1 ...... predicted state sequence. We define an error block as a maximal contiguous index interval [ that cannot be extended without including correctly classified point such that Yk, I [ Pk pt and Vk E [i P4 # Tk-
.PN) E S the
.p.....PN)
For an error block [i, j] in P. we define the atomicity A|. distinct states within R Based on A we introduce a novel typology of errors (illustrated in Fig. 2), such that each error block belongs exactly one error type. Our typology is as follows:
{ [kk [i,s]] } as the number of
Delay (A 1): The real and predicted states within [i.]] are constant, say and p respectively. Moreover, at least one block neighbor exists and satisfies 7:-1 Pi-1 p porrye TJ+1 Pj+1 p.
3

---

Ground truth
Ground truth
Ground truth
Ground truth
Delay
Isolated state
Transient state
Missing states
a) Delay
b) Isolation
c) Transition
d) Missing
Figure Ground truth (top) with four error examples below: delay, isolation, transition, and missing.
Isolation (A = 1): The real and predicted states within [i.5] are constant, say r and p respectively. Moreover, the error occurs in middle of constant real state, when f;-1 ry+1 Transition (A = 2): There are exactly two distinct real states within [i.j]] indicating the error spans
real state change.
Missing (A 2 3): Three or more distinct real states appear within [i, indicating omission of three or more real states.
This typology is important some errors might be less severe than others. Indeed, state boundaries are generally identified in a strictly binary manner, which may not suit some real-world applications. In practice, transitions between activities are often gradual rather than instantaneous. Thus, differences these transient periods. While some research propose to introduce gradual labelling [23]. simpler strategy would be to propose measures that are robust to labeling ambiguities for instance, deciding exactly where to place the boundary between walking and running, with the hypothesis that in such cases, error near real boundary (i.e., transient state or delay) is less severe than one that occurs in the middle of homogeneous region (i.e., missing state or isolated state).
between real and predicted states can arise from alternative interpretations
Based on the need to rank error types (as motivated above), we propose a set of properties that should be satisfied by any measure for state detection. These properties are designed to provide a more meaningful evaluation of state detection algorithms.
Desired Properties:
P1: The measure should be sensitive to the errors length, with larger errors leading to lower scores.
P2: The measure should account for the temporal structure, penalizing positions of errors differently. P3: The measure should be sensitive the type of error, with different penalties for different types.
P4: The measure should be interpretable and provide insights into the quality of the segmentation.
While these properties provide valuable guidance for the development and evaluation state detection measures, we emphasize that they not formal axioms and may not be strictly simultancously satisfied in all cases. For example, although Property P1 suggests that longer errors should lead to lower scores, the impact of error's length may depend its context, such whether it results from a delay or from an isolated error. In such cases, the score may be moderated by considerations of temporal position (Property P2) error type (Property P3). Thus, these properties should be viewed as guiding principles rather than rigid requirements.
2.3 Existing Measures and Limitations
While several measures have been adopted in the literature, each comes with set of assumptions and drawbacks, failing to catch some of the desired properties mentioned above. This section reviews these commonly used measures, discussing their strengths and limitations.
2.3.1 Change Point Detection Measures
Among the commonly used measures for change point detection, the F1 score is harmonic measure that combines precision and recall. Following previous work, we value margin tolerant F1 score, identifing the correct detections by matching predicted change points ground-truth -trth annotations within a given margin, while preventing double-counting of predictions by removing matched points after association. However, selecting the appropriate margin challenging. The example Fig. 3(a) illustrates two scenarios S1 and S2 in which the F1 score is 1, although S1 contains longer error than $2, thus failing to meet Property P1. margin parameter that a function of the time series length is often preferred, proposed in [14], where it is set to 1 1% of the time series length.

---

(a) Fi-score
(b) Covering score
(c) ARI
GT
Margin
Largest IoU
Delay
S1
Mangin
Largest IoU
Isolation
$2
Figure 3: Limitations of (a) F1, (b) Covering, and (c) ARI scores. For two different segmentations (S1 more accurate than S2 according to the ground truth GT), all measures return the same score.
The covering score, another commonly used measure, captures segment-level similarity rather than exact change point matching. Unlike the F1 score, which treats change points as discrete events, covering accounts for segment overlap. (IoU) scores for each segment in the ground truth, normalized by the number of segments. With π and P representing real and predicted state sequences, the Covering score calculated as follows:
defined as the average of the intersection over union
1 C= = N
Σ YER
r B Apl r
max PEP
However, the covering score can assign identical scores to segmentations that are qualitatively different, thus, failing to meet Property P1. Fig. 3(b) illustrates this limitation with two predicted segmentations that achieve the same covering score despite significant differences. This highlights the need for complementary measures that better capture segmentation quality.
2.3.2 State Detection Measures
State detection performance is most commonly evaluated with clustering- based measures [21, 20], such as the Adjusted Rand Index (ARI). the Normalized Mutual Information (NMI) and the Adjusted Mutual Information (AMT). In the rest of the paper, we will focus ART and we provide details on the additional clustering-based measures in the Appendix.
The computation of ARI is based on the Rand Index (RI) that computes the fraction of agrecing pairs (i.e., pairs that are either grouped or separated together) over the total number pairs. Formally, with R and P representing real and predicted state sequences and UR = {r:
E R} and Up = {P: : Ps E PJ the unique sets of states, we define the contingency matrix C = [nty] of size [Un| [Ur] with Mij = k= [rr Un( Pt= Urbi}, i.e., the number of observations at timestamp & that belong to state UR[ in the first state sequence R and Upli] in the second state sequence P. Finally, with [E[R] the expected Rand Index under random model, ARI is computed
as follows:
RI En ("E
RI - E ARI
(2)
1- E[R
As shown in the equations above, the Adjusted Rand Index (ARI) is sensitive to the number of matching temporal point pairs between segmentations. Thus, the total number of segmentation errors directly influences the ARI score, satisfying Property P1. However, clustering- ARI are inherently point-based and do not account for the position or type of segmentation errors. For example, Fig. 3(c) demonstrates this limitation: two predicted segmentations yield the same ARI score despite exhibiting markedly different segmentation error patterns (one delay versus one isolated error). Consequently, clustering- based measures fail to satisfy Properties P2 and P3.
measures like
3 WARI and SMS: Our Proposed Measures
As outlined the previous section, existing evaluation measures exhibit significant limitations (failing to meet either Property P1, P2 or P3). Moreover, these measures provide limited interpretabilityy making difficult for practitioners to understand accuracy scores pinpoint specific weaknesses and areas for improvement (failing meet Property P4). To address these shortcomings, propose
5

---

two state detection measures. The first one, WARI, consists of modified (weighted) version of the standard ARI, making it piftancc-to-boundan sensitive. The second one a new measure, namely SMS (State Matching Score), that identifies mapping between the predicted and real states, which then used to compute score based on the contexts and types of errors encountered according to the taxonomy defined in the previous section (Sec. 2.2).
3.1 Toward Position--Semsitiitt:: The Weighted Adjusted Rand Index
As mentioned in Sec. 2.3, the Adjusted Rand Index (ARI) treats all segmentation errors equally, However, segmentation errors near cluster boundaries are less critical than errors the cluster interior (i.e., Property P2). To account for this, define a weighted version of ARI, Weighted Adjusted Rand Index (WARI), based on the distances to change points. More specifically, this distance, called d,, is defined for each time step i and corresoonds to the distance from the nearest ground truth change point. We then define weight w; = 1+ a dj for each timestamp. & 2 is a user seee--aaaette,, set by default to 0.1 in the rest of the paper. For a > O, observations deep inside ground truth segments (i.e., with high de) are given more weight, and thus, more penalized wrongly predicted.
Weighted Contingeney Matrix: In the weighted setting described above, the contingency matrix (defined in Sec. 2.3.2) is adapted by replacing counts with weighted sums: Ray Ex,EU,ry Wkk- The weighted Adjusted Rand Index is then computed by using 7i values (and the corresponding total weighted sum of pairs) in Equation 2. Note that such weighted procedure can be applied to other clustering based measures, such as Normalized Mutual Information (NMI) and Adjusted Mutual Information (AMI). We provide more details in the Appendix.
Properties: WARI behaves like ARI with a boundary-aware lens. When a = 0, the weights collapse w; and WARI exactly coincides with ARI. As soon as > the measure starts to "prefer" boundary- adjjaet mistakes: points far from change points receive larger weights than those near them, so interior misclassifications are penalized more strongly, encoding the desired position sensitivity (P2). Yet the overall scale remains familiar--Wi reaches 1 under perfect agreement, drifts toward 0 for random labelings, and may become negative for strongly discordant pegmentationssppesevinnnn ARI's qualitative range and interpretation.
Sensitivity the position parameter a: Because (;(a) 1 a d; varies linearly in with d, E [0, Dmax], the weighted contingency and the resulting WARI vary smoothly with a. In particular, for two settings a and a' the score difference is bounded linearly: [WARI(a)
WARI(a')) I <
|ax a'), where I. depends only on the dataset via distances to boundaries and segment masses (details in Appendix). Practically, this yields robust behavior around the default a=0.1: moderate changes of a produce limited, predictable variations in the score. Larger emphasizes interior purity (harsher penalties far from boundaries), while smaller a increases boundary tolerance.
Algorithm 1 Optimal State Mapping
Require:
(ri,..., FN) and (p1.... .pN). .NN) 1: Compute unique sets UR = (r, : r, E R} and Up {p. p. E P}
The real and predicted state sequences
2: Compute cost matrix C' of size [Uel x [UR], such that, for Pa E Up (row i) and rw UR (column j), the negative overlap City follows:
C. - ---- P
3: Find Optimal Assignment: Apply the Hungarian algorithm [32] to C' to find mapping A from states in Up states in UR that minimizes the total cost (maximizes total overlap).
4: for all p. e Up not assigned by M do
P> Handle unassigned predicted labels 5: Set M(pu) to the smallest non-negative integer m such that m is not assigned by M.
6: end for
7: return Final mapping M.

---

Algorithm 2 State Matching Score (SMS)
Require: Real sequence R. prodiction P. mapping M, penalty weight w = { WMttlyy Wromsiion; Misolation, Mmiissing
1: P + (M(pi)....,
D Maa predictions to real states
2: Let he the of error blocks in P (cf. typology, Sec. 2.2).
3: for all b = [
4:
I + -
(block length)
5:
A + [rr i & S 5}l (atomicity)
6:
e - (delay, isolation, transition, missing) (determine error type)
7:
if E fisolation, transition) then find nearest real change points Uprev < i and bnext > j)
8:
+ 2 min(i Sprev:
bnexe
9:
(normalized distance to change point)
10:
end if
(11 + we),
delay,
11:
Pen(b)
((1 dw.),,
(isolation, transition),
((1 + (1
(we 1)1)
missing.
A
12: end for
return SMS
Pen
N BEB
3.2 Enhancing Interpretability: The State Matching Score
Whereas WARI takes into account the position of the errors (i.e., satisfying Property P2), the types of the errors not considered in the accuracy score, and the interpretability of the score is low (i.e., failing to meet Properties P3 and P4). To address these shortcomings. We introduce a novel interpretable and customizable measure, called the State Matching Score (SMS). The core idea relies on aligning the predicted and ground truth state sequences, taking into account the types and associated severity of errors made by the algorithm.
The State Matching Score (SMS) is computed through a two-stage process, detailed in Algorithm 1 and 2. First, an optimal mapping between predicted and real states is established using the Hungarian algorithm [32] on a cost matrix representing the negative overlap between unique states (Algorithm 1). This ensures that predicted state labels aligned with real state labels in way that maximizes overall agreement. Second, the State Matching Score itself is computed (Algorithm 2). This involves identifying error blocks in the mapped predicted sequence. classifying these errors according to the typology in Sec. 2.2, and assigning a penalty to each block based on its type, length, and context (i.e. distance real boundaries, atomicity). The final SMS is normalized score reflecting the overall quality of the state detection.
As shown in Algorithm 2, SMS incorporates penalty weights for different error types, allowing for customization to specific applications. For instance, in scenarios where reaction time is critical and false positives are tolerable, delays might be penalized more heavily than missing states. Despite this flexibility, the SMS exhibits robustness to the choice of these penalty weights. The overall score is primarily influenced by the total number of errors rather than the precise weight distribution.
Formal properties and guarantees. SMS designed be interpretable, predictable and robust to changes of its knobs. Intuitively, the score first reflects how much time is mislabelled (the total length of error blocks), and then applies controlled, interpretable refinements for error types and contexts.
all penalty weights are set to zero, SMS simply reduces to the fraction of correctly labelled time:
E
SMS 1
N'
where is the total length of error blocks and N the sequence length. With nonzero weights, since d E [0, 1] and A > (as defined above), the score remains tightly bounded:
(1 + Wmax) E
< SMS < 1
N
N
where Wmax is the largest penalty weight. Thus, weights can only modulate the score within explicit, predictable limits around the baseline 1 E/N.
7

---

Changing the weights moderately cannot swing the score wildly. For two weight settings w and w', the difference in scores is bounded by
E N
(SMS(w)= SMS(w')I _ ||0 w
When the overall error mass E/N is small-as desired for good Segmentations--SSS is provably stable to weight choices. We provide additional experiments measuring the robustness of SMS in the Appendix.
4 Experimental Evaluation
We now empirically evaluate the advantages of our proposed measures. In total, we consider
panel of 6 segmentation methods (F2USD [20], Time2State [21], HDP-HSMM [18], TICC [31], ClaSP [14] (used with kMeans clustering). and PaTSS [23]). We exclude HVGH [22] due to its poor performance [20, 21], and AutoPlait [33], which previous studies reported as non non-functional [20. 21]. In addition, consider benchmark of 5 datasets (PAMAP2 [15], USC-HAD [34], UCR-SEG [35], ActRecTuu [36], MoCap [37]) spanning various domains. Given the emphasis of this paper on state detection, particular attention is given to comparing WARI and SMS with ARI, i.e., the most commonly used measure in the literature. Finally, we provide an open-source implementation of measures and evaluation. Additional experimental setup details can be found in Appendix.
4.1 Evaluating the Evaluation Measures
We first design a synthetic experiment that evaluates sensitivity to error length, position, and type. The results, shown in Fig. 4, highlight distinct behaviors across the three measures (ARI, WARI, and SMS). First, all measures are sensitive to the error length (cf. Fig. 4(a)), exhibiting decreasing scores segmentation errors grow is length, across segmentations S1 to S9 (satisfying Property 1). Second, while WARI and SMS react to the position of the error pecn--zinniiigggggn isolated errors more heavily- remains insensitive (cf. Fig. 4(b)), assigning constant score regardless of the error's location (failing to meet Property 2). Finally, we assess sensitivity to error type, comparing measures behavior delay and transition error of same lenght (cf. Fig. 4(c)). While SMS assigns different scores to each case, both ARI and WARI return identical values, demonstrating their insensitivity to error type (failing meet Property 3).
We now present in Fig. 5 a qualitative comparison of segmentation results from E2USD and Time2State on a MoCap dataset time series. Traditional measures like ARI marginally favor E2USD, despite exhibiting clear isolated errors and less accurate segmentation overall. In contrast, Time2State produces more consistent segmentation, primarily with delay and transition errors. The proposed SMS, along with WARI, pick Time2State's output as the best segmentation. Specifically, SMS offers interpretable diagnostic of error types, feature lacking in conventional measures.
Public htapliiggttppooogooooooooo///aaa
(a) Evaluating Property
(b) Evaluating, Property 2
(+) Evaluating Property 3
61
61
G1
59
0.000
04
Mar emors
6315
0750
More Selatl
Mone inokted
XARI
J SMS
WARR
Figure 4: Synthetic data examples illustrating various error types and measure responses.
8

---

walking
jamping
Working
Hop ongin
ptreschiy
F2USD:
ARE: 0.30,
WARI: 0.86,
SMS: 0.77
according to
E2USD Segmentation Emor Diagnostic (produced by SMS)
Time2State:
ARI: 0.77.
WARI: 0.93,
SMS: 0.86
v Best nocording to WARI and SMS
Time2State Segmentation Error Diagnostic (produced by SMS)
Error
delay
trandition
missing
isalation
Figure Segmentation time series from the MoCap dataset using E2USD and Time2State.
(3) CLASP on UCRSEG
(a) SMS versus ARI for datasets and methods
(d) TICC PAMAP2
State Matching Score (SMS)
Guard Turk (01)
Spyroeadon
0.6
TICC UCRSEG
(c) PATNS on UCRSEG
Groand
Spgneraatiee
MoCap
Scancnation
0.0
PAMAP2
Index (ARD)
0.0
0.2
0.4
0.6
0.8
1.0
Figure 6: Comparison of ARI and SMS on real datasets collection.
More generally, we investigate how the proposed WARI and SMS measures compare to ARI on real-world data. Fig. 6 shows scatter plot of SMS versus ARI across the 5 datasets with the 6 segmentation methods (the comparison with WARI can be found in Appendix). While most points align along the diagonal (i.e., similar ARI and SMS scores), several points deviate significantly. In many cases, these deviations arise in settings with very few ground truth segments, where ART assigns low scores for predictions consisting of single segment. In contrast, SMS still assigns a proportional score based on how much of the smallest ground truth segment recovered. For example, Fig. 6(c) shows an time series from the UCRSEG dataset where TICC predicts single segment, resulting ARI R whereas SMS captures the partial match and yields a higher score. In other scenarios, such as Fig 6(b) and 6(e), SMS assigns more favorable scores than ARI due to its tolerance to temporal misalignment (e.g., delay errors), especially in cases where ground truth labels may themselves be subjective ambiguous. This illustrates that SMS can better capture meaningful segmentations despite imperfect annotations. However, as SMS is not adjusted for chance, it may overvalue simplistic segmentations (e.g., single segment) where error types are less relevant due to few states. As WARI (shown to be more accurate than ARI earlier) is adjusted for chance, we highlight the complementary nature of SMS and WARI, suggesting the interest in their joint usage.
4.2 Impact on State of the Art
We evaluate the relative performance of segmentation algorithms across datasets, using the pairwise Wilcoxon sign rank test, with a critical value of a = 0.05. Each time series is treated as an individual test instance. The corresponding critical diagrams are in Fig. 7 for ARI, WARI and SMS. Critical Diagrams for Fl and Covering can found in the Appendix.
Where we are: The algorithm rankings remain consistent, with the only notable exception being SMS on univariate datasets. In the univariate context, ClaSP systematically achieves the highest rank, aligning with previous works results [20, 21]. Regarding multivariate time series, Time2State

---

(a) Univariate dalasets
Adjinted Rand Index (ART)
Weighted Adjusted Rand Index (WAR))
Matching Score (SMS)
CLASP TIMESSTATE
TICC PATES
CLASP E2USD
HDP HSMM PATSS
CLASP
A000
A052.
HDF_ HSNM
TIME2STATE
Multivariate datasets
Adjusted Rand Index (AR3)
Weighted Adjusted index (WARD)
Matching
HDP HIMM
S10L TINE2STATE
EZUSD
HDP
CC
1991
EZUSD
TIMEXSTATE
HDP HISHI
100 TIME2STATE
EZUSD
Figure 7: Critical diagrams of state detection algorithms on (a) multivariate and (b) univariate datasets.
0.7
More isalation evors
(a) All datasets
More Missing ornors
Evor (1-ccore)
%
(b.1) ActReeTii
(b.2) MoCap
Error (1-score)
Legend:
ARI Error
0.4
(b.3) USC-HAD
(b.4) UCRSEG
SMS Error IDelay!
0.3
Brror (1 (1-sorr))
(Trnnstioon
0.2
SMS Error lisolation)
SMS Error (Missing)
TOMEZ5TATE
EXUs0
H09 Hote
clesp
tecc
ParSp
CLAb
caos resus
net
Horal
Hot
Figure 8: Error rate and error type contribution for (a) all datasets, and (b) per datasets.
outperforms other methods with statistically significant difference in all measures. We also observe that TICC and HDP-HSMM are often ranked last in both univariate and multivariate settings.
What is new: While previous studies typically conclude at the level of the analysis described above, we employ SMS to further explore performance comparisons across different types of errors. Fig. 8 depicts the errors (i.e., 1-Ccore) of each method, per dataset and measure, with the types of errors highlighted in the SMS bar. On average (Fig. 8(a)), we observe a significant distinction between (i) neural and probabilistic methods (such as Time2State, E2USD, and HDP-HSMM) which tend to produce more isolated errors, while (ii) other methods (such as ClaSP, TICC, and PaTSS) predominantly make missing and delay errors. However, the frequency and type of error vary significantly across datasets and methods, suggesting deep heterogeneity in segmentation behavior.
What is ahead: Beyond evaluation, the interpretable SMS framework suggests interesting research directions. The diversity in error types highlights opportunities for refining method selection, parame- ter tuning, and algorithm development. Specifically, error-type analysis guide the learning process and assess the parameter tuning step. For instance, the prevalence of isolated errors in Time2State, E2USD, and HDP-HSMM might be mitigated by adjusting parameters that control the number of clusters (e.g., the concentration parameter in the Dirichlet Process Gaussian Mixture Model used by Time2State). Conversely, for methods like ClaSP, TICC, and PATSS, which tend to produce missing errors, increasing the number of generated clusters could be beneficial (e.g., for ClaSP, by lowering the statistical test threshold for change point detection). Overall, error-type analysis can enhance both evaluation pipelines, as well as training, tuning and development processes.
Conclusion
We address a key gap in time series segmentation evaluation by formalizing typology of four distinct errors types, and proposing a set of desirable properties for evaluation measures. We introduce two new evaluation measures, WARI and SMS, that overcome major limitations of existing approaches and provide important novel insights. Such insights, open promising directions for error-aware model selection, development, tuning, and ensembling. Overall, this work contributes to interpretable, robust, and customizable tools to advance the evaluation and design of segmentation algorithms.
10

---

Acknowledgments and Disclosure of Funding
This work was funded in part by the French government under management of Agence Nationale de Recherche (ANR) as part of the "France 2030" program, reference ANR-23-IACL-O0M (PR[AI]RIE- PSAI). The authors are grateful to the CLEPS infrastructure from the Inria of Paris for providing resources and support.
References
[1] Anthony J. Bagnall, Richard L. Cole, Themis Palpanas, and Konstantinos Zoumpatianos. Data series management (dagstuhl seminar 19282). Dagstuhl Reports, 9(7):24-39, 2019.
[2] Themis Palpanas. Data series management: The road to big sequence analytics. SIGMOD Rec., 44(2):47-52, 2015.
[3] Themis Palpanas and Volker Beckmann. Report on the first and second interdisciplinary time series analysis workshop (ITISA). SIGMOD Rec., 48(3):36 2019.
[4] Shinan Liu, Tarun Mangla, Ted Shaowang, Jinjin Zhao, John Paparrizos, Sanjay Krishman, and Nick Feamster. AMIR: active multimodal interaction recognition from video and network traffic in connected environments. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 7(1):21:1-2::66 2023.
[5] Paul Boniol, Mohammed Meftah, Emmanuel Remy, Bruno Didier, and Themis Palpanas. denn/dcam: anomaly precursors discovery in multivariate time series with deep convolutional neural networks. Data-Centric Engineering, 4:e30, 2023.
[6] Adrien Petralia, Philippe Charpentier, Paul Boniol, and Themis Palpanas. Appliance detection using very low-frequency smart meter time series. In Proceedings of the 14th ACM International Conference on Future Energy Systems, e-Energy '23, page 314-225, New York, NY, USA, 2023. Association for Computing Machinery.
[7] Sylvain W. Combettes, Paul Boniol, Antoine Mazarguil, Danping Wang, Diego Vaquero-Rann Ramoss Marion Chauveau, Laurent Oudre, Nicolas Vayatis, Pierre-Paul Vidal, Alexandra Roren, and Marie-Martine Lefévre-Colau. Arm-CODA: A Data Set of Upper-limb Human Movement During Routine Examination. Image Processing On Line, 14:1-13. 2024.
[8] Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Discov,, (1(3):606666) 660, May 2017.
[9] John Paparrizos and Luis Gravano. k-shape: Efficient and accurate clustering of time series. SIGMOD Rec., 4561:69777,, June 2016.
[10] John Paparrizos, Yuhao Kang, Paul Boniol, Ruey S. Tsay, Themis Palpanas, and Michael J. Franklin. Tsb-uad: an end-to-end benchmark suite for univariate time-series anomaly detection. Proc. VLDB Endow., 15(8):1697-171) April 2022.
[11] Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly detection in time series:
a comprehensive evaluation. Proc. VLDB Endow., 1799)1179-11777, May 2022.
[12] Qinghua Liu and John Paparrizos. The elephant in the room: Towards reliable time-series anomaly detection benchmark. NeurIPS 2024, 2024.
[13] Patrick Schäfer and Ulf Leser. Motiflets: Simple and accurate detection of motifs in time series. Proceedings of the VLDB Endowment, 16(4):725-737, 2022.
[14] Arik Ermshaus, Patrick Schäfer, and Ulf Leser. ClaSP: parameter-free time series segmentation. Data Mining and Knowledge Discovery, 126-)0126 May 2023. Publisher: Springer Science and Business Media LLC.
[15] Attila Reiss. PAMAP2 Physical Activity Monitoring, 2012.

---

[16] Shaghayegh Gharghabi. Yifei Ding, Chin-Chia Michael Yeh, Kaveh Kamgar, Liudmila Ulanova, and Eamonn Keogh. Matrix Profile VIII: Domain Agnostic Online Semantic Segmentation at Superhuman Performance Levels. In 2017 IEEE International Conference on Data Mining (ICDM), pages 117-126, New Orleans, LA, November 2017. IEEE.
[17] Shohreh Deldari, Daniel V. Smith, Amin Sadri, and Flora Salim. ESPRESSO: Entropy and ShaPe awaRe timE-Series SegmentatiOn for Processing Heterogeneous Sensor Data. Proceed- ings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3):1224, September 2020. Publisher: Association for Computing Machinery (ACM).
[18] Masatoshi Nagano, Tomoaki Nakamura, Takayuki Nagai, Daichi Mochihashi, Ichiro Kobayashi. and Masahide Kaneko. Sequence Pattern Extraction by Segmenting Time Series Data Using GP-HSMM with Hierarchical Dirichlet Process. 2018 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4067-4074, Madrid, October 2018. IEEE
[19] Saachi Jain, David Hallac, Rok Sosic, and Jure Leskovec. MASA: Motif-Aware State Assign- ment in Noisy Time Series Data, June 2019. arXiv:1809.01819 [cs].
[20] Zhichen Lai, Huan Li, Dalin Zhang, Yan Zhao, Weizhu Qian, and Christian S. Jensen. E2Usd: Efficient-yer Efficient-yet-effecte yet-effective Unsupervised State Detection for Multivariate Time Series. In Proceed- ings of the ACM Web Conference 2024, pages 3010-3021, Singapore Singapore, May 2024. ACM. E2USD.
[21] Chengyu Wang, Kui Wu, Tongqing Zhou, and Zhiping Cai. Time2State: An Unsupervised Framework for Inferring the Latent States in Time Series Data. Proceedings of the ACM on Management of Data, 1(1):1-18, May 2023. Publisher: Association for Computing Machinery (ACM).
[22] Masatoshi Nagano, Tomoaki Nakamura, Takayuki Nagai, Daichi Mochihashi, Ichiro Kobayashi, and Wataru Takano. HVGH: Unsupervised Segmentation for High-Dimensional Time Series Using Deep Neural Compression and Statistical Generative Model. Frontiers in Robotics and
AI. 6, November 2019. HVGH.
[23] Louis Carpentier, Len Feremans, Wannes Meert, and Mathias Verbeke. Pattern-based time series semantic segmentation with gradual state transitions. Shashi Shekhar, Vagelis Papalexakis, Jing Gao, Zhe Jiang, and Mattco Riondato, editors, Proceedings of the 2024 SIAM International Conference on Data Mining, SDM 2024, Houston, TX, USA, April 18-20, 2024, pages 316-324. SIAM. 2024.
[24] Jushan Bai. Estimating multiple breaks one a time. Econometric Theory, 13(3)3315-352, 1997.
[25] R. Killick, P. Fearnhead, and I. A. Eckley. Optimal detection of changepoints with a linear computational cost. Journal of the American Statistical Association, 107(500)) December 2012. arXiv:1101.1 1438 [stat].
[26] Ryan Prescott Adams and David J. C. MacKay. Bayesian online changepoint detection, 2007.
[27] Mahtab Mirmomeni, Lars Kulik, and James Bailey. A Transferable Technique for Detecting and Localising Segments of Repeating Patterns in Time series. In 2021 International Joint Conference on Neural Networks (LJCNN) pages 1-10, Shenzhen, China, July 2021. IEEE
[28] Stefan Gaugel and Manfred Reichert. PrecTime: A deep learning architecture for precise time series segmentation in industrial manufacturing operations. Engineering Applications of Artificial Intelligence, 122: 106078, June 2023. Publisher: Elsevier BV.
[29] Yi Lu, Peng Wang, Bo Tang, Shen Liang, Chen Wang, Wei Wang, and Jianmin Wang. GRAB: Finding Time Series Natural Structures via A Novel Graph-based Scheme. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 2267-2272, Chania, Greece, April 2021. IEEE
[30] Shima Imani and Harsh Shrivastava. arXiv:2303.1 1647 [cs].
Are uGLAD? Time will tell!, October 2024.
12

---

[31] David Hallac. Sagar Vare, Stephen Boyd, and Jure Leskovec. Toeplitz Inverse Covariance-Bast Based Clustering of Multivariate Time Series Data, May 2018,
[cs].
[32] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955.
[33] Yasuko Matsubara, Yasushi Sakurai, and Christos Faloutsos. AutoPlait: automatic mining of evolving time sequences. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pages
Snowbird Utah USA, June 2014. ACM.
[34] Mi Zhang and Alexander A. Sawchuk. USC-HAD: daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 1036-1043, Pittsburgh Pennsylvania, September 2012. ACM.
[35] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-MI. The UCR Time Series Classification Archive, October 2018.
[36] Andreas Bulling, Ulf Blanke, and Bernt Schicle. A tutorial human activity recognition using body-womn inertial sensors. ACM Computing Surveys, 46(3),1-33, January 2014. Publisher: Association for Computing Machinery (ACM).
[37] Carnegie Mellon University CMU Graphics Lab motion capture library.
13

---

A Technical Appendices and Supplementary Material
State Defection Related Work
The Time2State method utilizes an encoder with a custom loss for representation learning part, then clusters the time series using sliding windows. The E2USD algorithm, builds upon Time2State by targeting the high computational overhead hindering streaming applications and improves contrastive learning by better handling false negatives. HDP-HSMM adopts a non-parametrie Bayesian approach to model temporal dependencies and duration distributions, while TICC utilizes temporal consistency and clustering to identify recurring patterns. The ClaSP + k-means approach relies on the state-of- the-art change point detection algorithm ClaSP, followed by kMeans clustering for grouping similar segments. Finally, PaTSS leverages pattern matching techniques to detect and align state transitions.
Normalized Mutual Information (NMI)
The NMI measures the mutual information I(U:V) between two segmentations U and V. normalized by their entropies HU) and HVV):
21(U;V) NMI(U, H(U) + H(V)
Mutual information I(U; V) and entropies HU),, HV) are typically computed from the contingency matrix Rty- The NMI ranges from to 1, with indicating perfect agreement between the two segmentations.
Weighted Normalized Mutual Information (WNMI)
Similar to the ARI, the NMI treats all points equally, regardless of their position within segment. To incorporate temporal structure and boundary awareness, propose weighted version, the Weighted Normalized Mutual Information (WNMI).
Using the same weighting scheme as for WARI, where each observation my has a weight wi based on its distance the nearest boundary, we adapt the NMI calculation using the weighted contingency matrix nij-
Datasets
The datasets used in this study are publicly available and can be accessed through the following links:
PAMAP2 The Physical Activity Monitoring dataset a comprehensive collection of data aimed at facilitating research in human activity recognition. It comprises recordings from nine subjects, each equipped with three Inertial Measurement Units (IMUs) placed on the wrist the dominant arm, chest, and ankle, along with heart rate monitor. The dataset includes 18 different physical activities, such as walking, running, and various household tasks. [15]
USC-HAD The University of Southern California Human Activity Dataset designed to support research in human activity recognition using wearable sensors. It contains data from 14 subjects, wearing a single MotionNode sensor on the front right hip. The sensor captures tri-axial accelerom- eter and gyroscope data a sampling rate of 100 Hz. The dataset encompasses 12 activity classes, including walking, running, sitting, standing, and various transitional movements. [34]
UCR-SEG The UCR Time Series Archive [35] repository of time-series datasets widely used for evaluating algorithms in various domains, including human activity recognition. It offers a diverse collection of datasets with varying lengths and dimensions, encompassing a range of activities and sensor modalities.
ActRecTine Tut This dataset is designed to support research in human activity recognition using body- worn inertial sensors [36]. focuses on recognizing various hand gestures by analyzing data from inertial measurement units (IMUs) attached to the upper and lower arms. The dataset provides comprehensive framework for designing and evaluating activity recognition systems, detailing each component and offering best practices developed by the research community.
14

---

MoCap The CMU Graphics Lab Motion Capture Database [37] is comprehensive collection of motion capture recordings performed by over 140 subjects. It includes wide range of activities such walking, dancing, and various sports, providing free motion data for research purposes. The database offers downloadable motion files in various formats, supporting research in fields like computer graphics, animation, and human motion analysis.
The properties of the datasets used are detailed Table 1, as described in [20, 21J.
Table 1: Properties of the datasets used in the experiments.
Datasets
# States
# Channels
Length (k)
Time series # Segments
State duration (k)
MoCap
5~88
4.6-~10.6
6~111
0.4~~2.0
ActRecTut
6
31.4--32.6
42
0.02-5.1
PAMAP2
11
253~~408
10
18~25
2.0-40.3
USC-HAD
12
25.4~-56.3
70
12
0.6-13.5
UCR-SEG
2~3
2~:40
32
2~3
1~25
Table 2: Licenses of datasets used in experiments.
Dataset
License Usage Terms
PAMAP2
CC BY 4.0
USC-HAD
License found; encouraged for use by ubiquitous computing researchers
UCR-SEG
License not found; widely used research with least 1000 papers citing archive
ActRecTut
License found
MoCap
Free for research; commercial use allowed in products, but resale (even converted) prohibited
Experimental Setup
We evaluated each algorithm each dataset, using the same hyperparameters in [20]. We evaluated runtime and performance of each algorithm, comparing their results on different datasets. The experiments were conducted
standard hardware setup including an Intel Core i7 processor and 32GB of RAM. We set time limit of 24 hours for each dataset.
PaTSS and ClaSP did not run on PAMAP2, which contains sequences of 300,000 points long on average- about ten times longer than other datasets (see Table 1). They exceeded the runtime or memory limitations of our evaluation setup. While prior works reported results for these methods on subsampled versions PAMAP2, we argue that evaluating on truncated sequences significantly distorts the segmentation task.
Hyperparameters
Table 3 provides the hyperparameters used in the experiments.
15

---

Table Parameters for State Detection and Evaluation Measures
Component
Parameters
State Detection Methods Parameters
Provided config. ig. joon files in the config ig folder the repos- itory root.
Evaluation Measure Parameters
WARI Weight:
a = 0.1
SMS Weights:
Wdclay 0.1
U trnnsitionn 0.3
Wisolationn 0.8
Wmissing = 0.5
Table 4: Algorithm performance across datasets and measures (mean = std dev). 'x' indicates timeout or memory еπгог
Dataset
Measure
HDP-HSMM
E2USD
PaTSS
Time2State
TICC
ClaSP
0.03 + 0.00
0.03 + 0.01
0.06
0.01
0,01
0.07 + 0.00
0.08 0.00
Covering
0.27 + 0.00
0.49 0.07
0.66
0.74 0.10
0.33 - 0.01
0.41 = 0.03
ARI
0.71 + 0.14
0.54
0.73
178:000 0.09
0.37
0.31 = 0.05
ActRecTut
NMI
0.70 0.08
0.64
0.74
0.53 0.75
0.01
WARI
0.67
0.17
0.83 0.1
0.51
0.44
WNMI
0.60
0.56 0.04
0.76
0.68 0.05
0.62
0.47 = 0.01
SMS
0.11
0.58 0.11
0.22
0.80
0.42
0.35 0.10
Fl
0.15
0.18 1 0.06
0.14
0.19 0.04
0.21 = 0.07
0.25 + 0.05
Covering
0.11
0.75 + 0.14
0.14
0.50 0.25
0.65 + 0.08
0.74 0.16
0.72 + 0.15
ARI
0.60
0.71 + 0.18 0.18
0.57
+.7++0.0 0.20
0.58 + 0.15
0.65 + 0.26
0.61 + 0.16
0.16
MoCap
NMI
0.09
0.73
0.68
0.66 0.12
0.68 0.26
0.11
WARI
0.14
0.82 0.20
0.65
0.75 0.14
0.75
0.68 = 0.19
WNMI
0.09
0.77 + 0.12
0.72 0.12
0.70 0.10
0.72
0.74 0.12
SMS
0.14
0.77 + 0.13
0.63
0.17
0.70 0.09
0.76 0.15
0.73 0.11 0.11
FI
0.10
0.18 + 0.10
0.05 + 0.05
0.22
0.54 + 0.11
0.59
A.59:0.100
Covering
0.12
0.20 0.24
0.44
0.67 = 0.20
0.79 0.19
ARI
0.12
0.36
0.15
0.30
0.16 0.30
0.59
UCRSEG
NMI
0.20
0.49
0.17
0.20
0.17 0.30
0,62 0.29
WARI
0.11
0.40
0.170025
0.22
0.18 0.33
0.65 + 0.36
WNMI
0.18
0.41
0.18 0.23
0.40 0.28
0.14 0.25
0.59 + 0.33
SMS
0.28
0.48 + 0.22
0.50 0.16
0.57 = 0.24
0.60 = 0.18
0.82 0.17
0.08 + 0.03
0.14 0.04
0.08
0.04
0.14 0.03
0.15 0.03
Covering
0.70
0.49
0.66
0.60 + 0.10
0.70 0.08
ARI
0.48 0.06
0.27 0.13
0.72 0.10
0.38
0.64 0.12
USC-HAD
NMI
0.79
0.16
0.83 0.04
0.69 0.10
0.81 = 0.06
WARI
0.13
0.37 0.13
0.81 0.12
0.52
0.73 0.15
WNMI
0.66 + 0.07
0.75
0.17 0.49
0.77 0.05
0.65 0.11
0.75 0.06
SMS
0.08
0.66 0.09
0.28
0.19
0.71 0.07
0.44
0.67 0.10
Fl
0.02 - 0.04
0.02
X
0.03 1 0.04
0.14 0.16
X
Covering
0.05 1
0.3
0.37 1 0.06
0.11
X
ARI
0.28+00.0 0.07
0.32+ 0.10
0.31 0.10
0,29 0.14
PAMAP2
NMI
0.52 0.09
160+0.10
0,59
0.55 + 0.07
WARI
0.42
0.50
0.49+
0.43 0.20
WNMI
0.60
0.65 0.19
0.65
0,66 0.14
SMS
0.53 + 0.07
0.54 + 0.16
0.53
0.52 + 0.16
16

---

Metries Implementation
The ARI and NMI are calculated using the sklearn library in Python, which provides efficient implementations of these measures. The Fl-score and covering scores are computed using custom implementation, adapted from TSSB code.
WARI ARI
The link between WARI and ARI is displayed Fig. 9.
0.8
1981
Dataset
0.2
USC HAD
UCBSEG
ActRecTut
MoCap
HAMAP2
0.0
0.2
0.6
0.8
1.0
Figure 9: WARI against ART score of each dataset.
Critical Diagram for Change Point Detection
Figure 10 depicts the critical diagrams for F1 score and Covering score for both univariate and multivariate time series datasets.
F1
HDP HSMM
4.4044
1,6755
TICC E2USD
PATSS HDP HSMM E2USD
3.812
1.3750
CLASP
TIME2STATE
2.9840
1.9255
TICC
3,9688
3.9175
TIME2STATE
COVERING
COVERING
HDP HSMM
552562
CLASP
HSMM 3.0574
1
E2USD
PATSS
5,2250
5.37
TICC
TICC
1.9362
TIME2STATE
E2USD
3,6250
2.1582
TIME2STATE
(a) Multivariate
(b) Univariate
Figure 10: Critical diagrams of state detection algorithms used as change point detectors. (a) Multivariate results. (b) Univariate results.
SMS Robustness Evaluation
As shown in Algorithm 2, SMS incorporates penalty weights for different error types, allowing for customization specific application requirements. For instance, in scenarios where reaction time is
17

---

0.08
0.08
0.06
0.06
0.04
0.04
0.02
0.02
TICC
CLASP
PATSS
TIME2STATE
E2USD
HDP HSMM
UCRSEG
ActRecTut
USC-HAD
MoCap
PAMAP2
(a) Algorithm
(b) Dataset
Figure 11: SMS variability (std) across 100 runs with random uniform penalty weights in [0, 1]
critical and false positives are tolerable, delays might be penalized more heavily than missing states. Despite this flexibility, the SMS exhibits robustness to the specific choice of these penalty weights. The overall score is primarily influenced by the total number of errors rather than the precise weight distribution.
To demonstrate this, evaluated all segmentations across the five datasets and six algorithms, using randomly assigned weights for each error type (drawn uniformly from [0, over 100 runs). As depicted in Fig. 11, the resulting score distributions showed limited variability, with average standard deviation 0.03169. This indicates that while parameter tuning can refine the distinction between error types, the fundamental performance ranking remains largely consistent.
Quantitative Evaluation of Error Types
The count of each error type for each method across datasets using the SMS as evaluation measure is displayed in Fig. 12.
ActRocTLLt
McCap
USC HAD
158
218
1.5K
500
5k
THME25TATE
ClaSp
Icc
CLASe
rams
HDP HoMe
Aams
UCRSEG
PAMAP2
151k
52k
E300
10P roue
CLAS
nice
raM TORL
CL4SP
Delay
Transition
so lation
Missing
Figure 12: Count each error type for each method across datasets using the SMS as evaluation measure
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims are we (i) review the literature and propose a typology of segmen- tation errors (cf. Sec. 2.1-2.2), (ii) highlight limitations of existing evaluation measures (cf. Sec. 2.3), (iii) introduce two new measures-WAR and SMS-with theoretical justifica- tions (cf. Sec. 3), (iv) empirically show their advantages over existing metries (cf. Sec. 4.1),
and (v) demonstrate their ability to reveal new insights into segmentation performance (ef. Sec. 4.2).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the work consist in the limitations of the proposed ap- proaches: (i) WARI not meeting Property 3 and 4 (ef. Sec. 4.1, Fig. 4), (ii) SMS being adjusted and leading to high score discrepencies when compared to ARI (cf. Sec. 4.1, Fig. 6), and being parameter dependant (ef. Sec. 3.2).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
19

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification:
The main results necessitate only the proposed measures implementation, that are described and can be re-implemented from information in the main text an Appendix: (i) WARI is described in Sec. 3.1, and (ii) SMS is explained with pseudo code (cf. Algo. 1 and Algo. 2).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data and documented code are available with open access from the url provided in the footnote in Sec. 4.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ public/gun ConideesoooeSuumsssiii..c
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / httpppscccppbbbccccpcceeee hewbbiiccguiieetee des/ htte///ooeSuumiiseeeeee ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All experimental details are given in main text (cf. Sec. 4) or in the appendix in case of missing space. Hyperparameters are given in the main text (as well as in the appendix) for WARI (cf. Sec. 3.1), while parameters for SMS are given in Appendix (cf. Table. 3).
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
21

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The standard deviation of the main experiment are given in appendix. The statistical test used is defined with the critical value used and critical diagrams with statistical test values.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides details about the hardware setup used, and the limitations (time and memory) set for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerripp.eeeee
ines?
Answer: [Yes]
Justification: Human activity recognition datasets are being used from previous avaible public archives.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
22

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact in the work performed. This work proposes novel evaluation framework and measures for time series segmentation algorithms. As methodological contribution, it does not involve the collection of human data, deployment in user-facing systems, or direct application to high-stakes domains. The proposed metrics aim to improve the rigor and interpretability of algorithm evaluation. Potential societal impact is indirect and depends entirely how these evaluation measures are applied in downstream tasks. We do not foresee any immediate negative societal consequences from this research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any models or datasets with high risk of misuse. introduces evaluation measures for time series segmentation and operates entirely on publicly available datasets under standard research licenses. The contributions are methodological and do not pose foreseeable risks requiring specific safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
23

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licenses usage terms given for each dataset in Appendix (ef. Table. 2). When the exact license is not found, usage terms from the accompanying paper are provided.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The supporting code of the paper provided and documented in the repository linked in footnote in Sec. 4.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There were no crowdsourcing experiments nor research with human subjects conducted in this work.
Guidelines:

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There were studies with human subjects conducted in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM usage is not an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: :ttp:://neuri.eeeee htt.://ps...ccooeeeeeeeee/////// for what should or should not be described.
25


Reviews:
[{"review_id": "Vw3vskzh1m", "reviewer": "Reviewer_nH5j", "summary": "This paper introduces two evaluation measures, WARI (Weighted Adjusted Rand Index) and SMS (State Matching Score), that \"provide a more accurate assessment of segmentation quality and uncover insights, such as error provenance and type, that are inaccessible with traditional measures.\"", "questions": "Please see weaknesses.", "limitations": "Yes. The paper has provided a thorough discussion about its the limitations of its proposed methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and logically structured, and the proposed metrics offer interpretable diagnostics by identifying and categorizing fundamental segmentation error types, which enhances comparative analysis across methods. The experimental results on both synthetic and real-world datasets further support the practical usefulness of the proposed metrics, particularly SMS, which allows error-type-aware evaluation and facilitates downstream model analysis and tuning.\n\n Weaknesses:\n\nThere are still some limitations to be addressed. First, the authors compare their proposed metrics only with traditional evaluation measures (e.g., ARI, F1, and Covering) but omit comparisons with recent advancements in segmentation evaluation. This omission weakens the empirical rigor of the study. Including such baselines would better contextualize the work within the current landscape of metric design. If directly comparable methods are unavailable, discussing related approaches and explicitly differentiating their scope and focus in the related work section would still be valuable.\nAdditionally, while the approach is practical and engineering-driven, it would benefit from deeper theoretical analysis or formal guarantees to strengthen the methodological foundation and generalizability of the proposed metrics.", "was_revised": false, "final_justification": "", "rebuttal": "> **The authors compare their proposed metrics only with traditional evaluation measures (e.g., ARI, F1, and Covering) but omit comparisons with recent advancements in segmentation evaluation.**\n> \n>\n> Thank you for this insightful comment. A key motivation for our work is the observation that, despite significant advancements in segmentation algorithms, the methods for their evaluation have not evolved accordingly. Our literature review confirms that even the most recent state-of-the-art papers on time series segmentation rely exclusively on traditional metrics. For state detection, methods such as Time2State (Wang et al., 2023), E2USD (Lai et al., 2024), and the recent CLaP preprint (Ermshaus et al., 2025) use ARI, AMI, or NMI. Similarly, for change point detection, the most recent algorithms like ClaSP (Ermshaus et al., 2023) are evaluated using F1 and Covering. This highlights a critical and persistent gap in the field, which our work directly aims to address. We will clarify this point in the related work section to better contextualize our contribution. However, if we have overlooked recent advancements in evaluation metrics, we would be grateful if the reviewer could share the relevant references.\n\n> **If directly comparable methods are unavailable, discussing related approaches and explicitly differentiating their scope and focus in the related work section would still be valuable.**\n> \n> This is a valuable point. While we did not find direct counterparts to WARI and SMS for time series segmentation, we agree that discussing metrics from related domains could strengthen the paper. We will expand our related work section to include a discussion of evaluation metrics from other sequential data domains (e.g., anomaly detection) and differentiate their objectives and methodologies from our approach, thereby better situating our work within the broader landscape of evaluation metric design.\n\n> **While the approach is practical and engineering-driven, it would benefit from deeper theoretical analysis or formal guarantees.**\n> \n> Thank you for raising this important point. We can indeed establish several formal properties for SMS that guarantee its robustness and bounded behavior.\n>\n> According to its definition in Algorithm 2, SMS is calculated as $\\text{SMS} = 1 - (E+P)/N$, where $N$ is the time series length, $E$ is the total length of all erroneous segments, and $P$ is the sum of penalties applied to these segments. The penalty term $P$ is a function of user-defined weights, allowing for application-specific adjustments. When all weights are set to zero, the penalty term vanishes, and SMS simplifies to $1 - E/N$, which is equivalent to segmentation accuracy (the fraction of correctly labeled points). This provides a clear and interpretable baseline for the score.\n>\n> The influence of the penalty weights on the final score is bounded and predictable. Formally, the SMS score is bounded by $1 - \\frac{(1+w_{\\max})E}{N} \\le \\text{SMS} \\le 1 - \\frac{E}{N}$, where $w_{\\max}$ is the maximum user-defined weight. This shows that the score is fundamentally anchored by the segmentation accuracy, with the weights providing a controlled adjustment. Furthermore, we can demonstrate that the metric's sensitivity to changes in the weight configuration is directly proportional to the total error size $E$.\n>\n> Specifically, the absolute change in the SMS score is bounded by $\\lVert w - w' \\rVert_\\infty \\cdot \\tfrac{E}{N}$, where $\\lVert w - w' \\rVert_\\infty$ is the maximum change across all weights. This property ensures that for a reasonably accurate segmentation (i.e., small $\\tfrac{E}{N}$), the choice of weights has a limited and bounded impact on the score, preventing arbitrary fluctuations and ensuring that the evaluation remains robust.\n>\n> A brief justification for these properties is as follows. Given $Err$ as the set of erroneous segments, we have $E=\\sum_{e \\in Err} |e|$ and $P= \\sum_{e \\in Err} |e|\\, w_e \\, \\operatorname{context}(e)$, where both the weight $w_e$ and the context term $\\operatorname{context}(e)$ are in $[0, 1]$. This leads to the bound $0 \\le P \\le w_{\\max} E$. Substituting this into the SMS definition gives the above presented bounds.\n>\n> Regarding the sensitivity analysis, let $w$ and $w'$ be two weight vectors. The difference in scores is $|\\text{SMS}(w) - \\text{SMS}(w')| = |P(w') - P(w)|/N$. The change in total penalty is bounded by $|P(w) - P(w')| \\le \\sum_{e \\in Err} |e| \\cdot |w_e - w'_e|$\n>\n> $\\le \\lVert w - w' \\rVert_\\infty\\sum_{e \\in Err} |e| = \\lVert w - w' \\rVert_\\infty E$. Combining these gives the sensitivity bound $|\\text{SMS}(w) - \\text{SMS}(w')| \\le \\lVert w - w' \\rVert_\\infty (E/N)$."}, {"review_id": "3OIJCTPzka", "reviewer": "Reviewer_fvnY", "summary": "This paper proposed two evaluation measures, WARI and SMS for time series segmentation.\nCompared with existing approaches, the proposed method can model the length, positions, and types of error.", "questions": "1. Have you considered formalizing theoretical properties of SMS and WARI, such as consistency or convergence guarantees?\n\n2. How does WARI fundamentally differ from other distance-sensitive clustering metrics? Are there scenarios where WARI behaves non-trivially compared to ARI with pre-alignment?\n\n3. Can you provide examples or experiments where using SMS as a selection criterion leads to improved model performance or better real-world segmentation?\n\n4. Could SMS or WARI be integrated into model training objectives or hyperparameter tuning workflows?", "limitations": "1. SMS is not chance-adjusted, which may lead to misleadingly high scores in trivial segmentations.\n\n2. The choice of penalty weights in SMS can influence results, although some robustness is demonstrated.\n\n3. Lack of model-guided improvement loop: it remains unclear whether these metrics can be used to improve segmentation algorithms in practice.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### [Strengths]\n1. The motivation of this work is clear. It sounds reasonable that four desired properties can help more meaningful evaluation of time series segmentation.\n2. The organization of paper is clear and the writing is easy to understand.\n\n### [Weaknesses]\n1. The paper lacks deep theoretical analysis or formal guarantees for WARI and SMS (e.g., monotonicity, consistency, convergence).\n2. WARI is a relatively straightforward modification of ARI with time-based weights, which may be viewed as incremental rather than fundamentally novel.\n3. While the proposed metrics diagnose segmentation quality, it is unclear how they lead to improved model performance or better real-world decision-making.", "was_revised": true, "final_justification": "I agree with the response of authors on my concerns for improving this work.", "rebuttal": "> **Have you considered formalizing theoretical properties of SMS and WARI, such as consistency or convergence guarantees?**\n>\n> Thank you for raising this important point. We can indeed establish several formal properties for both WARI and SMS that guarantee their consistency and bounded behavior.\n>\n> **WARI Properties:**\n> WARI is a principled extension of the Adjusted Rand Index (ARI). It inherits all of ARI's fundamental properties, including adjustment for chance and a fixed range (typically close to 0 for random agreement and 1 for perfect agreement). WARI's behavior is controlled by a single parameter, $\\alpha$, which modulates the weight assigned to each data point based on its distance to the nearest true segment boundary.\n>\n> -   **Consistency with ARI:** When $\\alpha=0$, all points are weighted equally, and WARI behaves as the standard ARI.\n> -   **Asymptotic Convergence:** As $\\alpha \\to \\infty$, the weight of each point becomes dominated by its distance from the boundary. We can show that the $\\alpha$ terms cancel out in the WARI formula, causing the score to converge to a stable value. This limit corresponds to an ARI computed using a contingency matrix where each point's contribution is weighted purely by its distance to the nearest boundary. This ensures that the metric's behavior is stable and predictable even for large values of $\\alpha$. We have observed this convergence in practice, with the WARI score stabilizing for large $\\alpha$.\n>\n> **SMS Properties:**\n> We have established several formal properties for SMS that guarantee its robustness and bounded behavior.\n>\n> According to its definition in Algorithm 2, SMS is calculated as $\\text{SMS} = 1 - (E+P)/N$, where $N$ is the time series length, $E$ is the total length of all erroneous segments, and $P$ is the sum of penalties applied to these segments. The penalty term $P$ is a function of user-defined weights, allowing for application-specific adjustments.\n>\n> -   **Interpretability and Baseline:** When all weights are set to zero, the penalty term $P$ vanishes, and SMS simplifies to $1 - E/N$, which is equivalent to segmentation accuracy (the fraction of correctly labeled points). This provides a clear and interpretable baseline for the score.\n> -   **Boundedness and Stability:** The influence of the penalty weights on the final score is bounded and predictable. Formally, the SMS score is bounded by $1 - \\frac{(1+w_{\\max})E}{N} \\le \\text{SMS} \\le 1 - \\frac{E}{N}$, where $w_{\\max}$ is the maximum user-defined weight. This shows that the score is fundamentally anchored by the segmentation accuracy, with the weights providing a controlled adjustment.\n> -   **Robustness to Weight Selection:** We can demonstrate that the metric's sensitivity to changes in the weight configuration is directly proportional to the total error size $E$. Specifically, the absolute change in the SMS score for two weight vectors, $w$ and $w'$, is bounded by $\\lVert w - w' \\rVert_\\infty \\cdot (E/N)$. This property ensures that for a reasonably accurate segmentation (i.e., small $E/N$), the choice of weights has a limited and bounded impact on the score, preventing arbitrary fluctuations and ensuring that the evaluation remains robust. An experimental evaluation for these properties is provided in the appendix (p.14)\n> We will add a detailed discussion of these theoretical properties for both WARI and SMS, along with their formal proofs, to the appendix of the revised manuscript.\n\n> **How does WARI fundamentally differ from other distance-sensitive clustering metrics? Are there scenarios where WARI behaves non-trivially compared to ARI with pre-alignment?**\n> \n> Thank you for this insightful question. To the best of our knowledge, no distance-sensitive metrics are available for evaluating time series segmentation. Our literature review shows that even recent state-of-the-art methods (e.g., Time2State (Wang et al., 2023), E2USD (Lai et al., 2024), and the recent CLaP preprint (Ermshaus et al., 2025)) rely on standard, non-distance-sensitive metrics like ARI, AMI, and NMI. A key challenge in this context is that predicted state labels are arbitrary and do not align with ground-truth labels, making direct, point-wise distance calculations meaningless without a prior matching step. While metrics for time series clustering exist, they are not directly applicable to segmentation evaluation due to this label permutation issue. It is to fill this gap that we proposed WARI, as an adaptation of a non-distance-sensitive measure, and SMS, a native distance-based measure that relies on label pre-alignment.\n> \n> ARI is permutation-invariant, meaning its final score is independent of the specific numerical labels assigned to states. Therefore ARI provides the same score with or without pre-alignment, and we can refer to the examples provided in the paper to answer your specific question. We provide an example of such a scenario, where WARI behaves non-trivially compared to ARI, in Figure 5, which is discussed in detail in Section 4.1. In this example, we present two non-trivial segmentations, S1 and S2. While S2 is qualitatively superior to S1 (based on the desirable properties P1-P4 outlined in Section 2.2), the standard ARI metric incorrectly assigns a higher score to S1. In contrast, our proposed WARI metric correctly favors S2, demonstrating its ability to capture nuanced temporal errors that ARI overlooks.\n\n> **Can you provide examples or experiments where using SMS as a selection criterion leads to improved model performance or better real-world segmentation?**\n> \n> Thank you for this question, pointing towards metric's practical utility. While we do not present a dedicated experiment on downstream task performance, our paper provides the foundation and examples for how SMS can be used as a selection criterion to achieve better real-world segmentation outcomes.\n>\n> The primary utility of SMS in model selection comes from its detailed error diagnostic. Unlike monolithic scores, SMS provides a transparent breakdown of every error, detailing its type, size, position, and associated penalty. This diagnostic power allows a practitioner to move beyond a simple \"which model is better?\" to \"how is this model better, and do its strengths align with my application's needs?\"\n>\n> For instance, Figures 7 and 8 in our paper serve as a clear example of this. They provide a detailed breakdown of algorithm performance, highlighting the different error profiles each method exhibits. A practitioner can use this diagnostic to select the algorithm that best aligns with their specific requirements. For example, in a medical application where failing to detect a critical state (a \"missing\" error) is far more dangerous than a slight timing misalignment (a \"delay\" error), a user could select the model that minimizes \"missing\" errors, even if its overall score is slightly lower than a competitor's. This choice, guided by the SMS diagnostic, directly leads to a better and safer real-world segmentation outcome.\n>\n> Furthermore, under an i.i.d. assumption, a practitioner could evaluate several algorithms on a small, labeled subset of their data. By analyzing the error types with SMS, they could then select the most suitable algorithm for their specific use case and apply it to the larger, unlabeled dataset, ensuring the chosen model's behavior is optimized for what matters most in their domain.\n\n> **Could SMS or WARI be integrated into model training objectives or hyperparameter tuning workflows?**\n> \n>This question highlights a key direction for future work, which we discuss in Section 4.2. The integration of SMS and WARI into model development workflows can be viewed in two distinct contexts:\n>\n>1.  **Supervised and Semi-Supervised Settings:** For tasks like model selection or hyperparameter tuning where a labeled validation set is available, both SMS and WARI can be directly used as evaluation criteria. As discussed in our response to the previous question, a practitioner can select the model or hyperparameters that yield the best SMS/WARI score on the labeled data, tailored to the specific error types relevant to their application. This is a straightforward and highly practical application of our metrics.\n>\n>2.  **Unsupervised Settings:** Integrating these metrics directly into the training objective of an unsupervised model (i.e., without a ground truth) is more challenging. Since SMS and WARI are extrinsic measures that require a reference segmentation, they cannot be used as a loss function in a purely unsupervised context. Adapting our framework for such scenarios is currently our focus. An option we are considering is to use surrogate internal metrics (derived from WARI and SMS), leveraging the model centrality hypothesis, as has been proposed for related time series tasks such as outlier detection (Ma et al., 2023) and anomaly detection (Li et al., 2022). Investigating the applicability of such unsupervised model selection frameworks to time series segmentation, within the WARI/SMS scope is a key direction of our future work."}, {"review_id": "hfPbW5DnUN", "reviewer": "Reviewer_kyud", "summary": "This paper introduces two metrics for evaluating the quality of time series segmentations, addressing key limitations of existing evaluation methods, namely their inability to capture segmentation quality, differentiate between error types, or provide meaningful interpretability.\nThe first metric, WARI (Weighted Adjusted Rand Index), extends the traditional Adjusted Rand Index by incorporating the temporal positions of segmentation errors, enabling it to distinguish between early, late, or localized mismatches. The second metric, SMS (State Matching Score), identifies and scores four types of segmentation errors (delay, isolation, transition, and missing segments) and allows users to assign application-specific weights to each error type. The proposed metrics are empirically validated across multiple datasets and segmentation methods, comparing their diagnostic power to existing evaluation metrics.", "questions": "1. Hungarian matching applicability: The use of Hungarian matching in the evaluation framework appears well-suited for self-supervised or unsupervised segmentation settings, where the predicted states may not align with ground-truth labels. However, for supervised segmentation methods that learn a direct mapping to specific state labels, is this matching still necessary? Could it distort the evaluation if exact label alignment is already learned?\n\n2. Comparison of SMS and ARI scores: Figure 6 presents a clear and helpful visualization of how SMS compares to ARI. However, it consistently shows SMS scores as higher than ARI, which raises questions about potential score calibration or scaling differences. Is this overestimation expected? How should it be interpreted, especially since in other figures (e.g., Figure 8), SMS appears lower than ARI in many cases? Clarifying this would help with understanding how to read and compare these scores across experiments.", "limitations": "While the paper does not explicitly discuss limitations, I believe it's important to address a main concern, which is the sensitivity of the proposed metrics to user-defined weighting variables and weights. One of the strengths of this work is its potential to become a standard evaluation metric that others can use to benchmark and compare segmentation methods. However, the flexibility to customize weights for different error types ($\\alpha$ in WARI and error type weights in SMS), while valuable for application-specific analysis, also opens the door to overfitting the metric to favor particular methods.\n\nThis raises a concern: if weighting is tuned to benefit a specific model or use case, comparisons across methods may become less meaningful. In such cases, the metric's value may shift more toward being a diagnostic tool rather than a consistent benchmarking standard. A deeper discussion of this trade-off, and perhaps guidance on default weight settings or best practices for fair comparison, would be a valuable addition to the paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper thoroughly analyzes the problem of evaluating segmentation in time series and clearly articulates the properties that distinguish good from bad segmentations. This results in a detailed and structured framework for analyzing segmentation quality.\n2. I believe this work has strong potential, especially given the lack of robust existing solutions, and it could ultimately become a standard benchmark metric for time series segmentation.\n3. The visuals, result descriptions, and analysis are excellent. The use of diverse examples is particularly effective in motivating the problem and demonstrating how different metrics respond to segmentation errors.\n\n\nWeaknesses:\n\n1. My major concern is that the proposed scores—especially SMS—are highly sensitive to the weighting scheme. While this flexibility is useful for application-specific evaluation, it becomes a downside when the metric is used for benchmarking. There should be at least some guidelines or constraints on how weights are selected. For example, do weights exhibit linear behavior—i.e., if \"delay\" is twice as important as \"missing segment,\" should its weight simply be doubled? Even with that assumption, the choice of weights still feels somewhat arbitrary. The paper would benefit from a sensitivity analysis or guidance on recommended weighting strategies for common use cases.\n2. To establish these metrics as state-of-the-art tools in the field, the paper would benefit from a broader set of experiments. While the selected datasets and models are solid, I recommend comparisons against basic metrics such as error rate or MSE, as well as sensitivity analyses for the proposed metrics. One particularly compelling experiment would be to show that using the proposed metric for model selection leads to better downstream performance. This would demonstrate that the metric not only captures meaningful differences but is also useful in practice—especially since much of the current analysis remains qualitative or relative to metrics with known limitations.\n3. Typo in algorithm: Line 8 in Algorithm 2 assigns the same weights to all components. I assume these are intended to be different weights, so this should be corrected.", "was_revised": true, "final_justification": "I believe this paper is tackling and interesting and important problem. Overall, the paper was well-written and easy to follow. I originally had some concerns, but the rebuttal was very helpful in resolving those.", "rebuttal": "> **Hungarian matching applicability: The use of Hungarian matching in the evaluation framework appears well-suited for self-supervised or unsupervised segmentation settings, where the predicted states may not align with ground-truth labels. However, for supervised segmentation methods that learn a direct mapping to specific state labels, is this matching still necessary? Could it distort the evaluation if exact label alignment is already learned?**\n> \n> Thank you for the insightful question. Indeed, in a fully supervised setting, where a model is trained to predict specific, pre-defined state labels, the Hungarian matching step is not necessary. Our framework can be easily adapted for this scenario by bypassing the state alignment step. This is achieved by modifying a single line in Algorithm 2 (Line 1 is replaced by $\\tilde{P} \\gets P$), which ensures that the predicted labels are directly compared against the ground-truth labels. The rest of our formalism, including the error typology and weighting scheme, remains fully applicable and provides the same diagnostic power in both supervised and unsupervised settings.\n\n> **Comparison of SMS and ARI scores: Figure 6 presents a clear and helpful visualization of how SMS compares to ARI. However, it consistently shows SMS scores as higher than ARI, which raises questions about potential score calibration or scaling differences. Is this overestimation expected? How should it be interpreted, especially since in other figures (e.g., Figure 8), SMS appears lower than ARI in many cases? Clarifying this would help with understanding how to read and compare these scores across experiments.**\n> \n> Thank you for this keen observation. The two metrics operate on fundamentally different principles. ARI is a combinatorial, pair-counting metric where a single misclassified point can disproportionately impact the score by affecting its relationship with all other points ($N-1$ pairs). In contrast, SMS is a segment-based metric calculated as $1 - (E + P) / N$, where $N$ is the time series length, $E$ is the total length of all erroneous segments, and $P$ is the sum of penalties applied to these segments. Its score is anchored by the proportion of correctly classified points, with nuanced adjustments based on the type, size, and context of error segments. This design means that the impact of an error on the SMS score is more nuanced as localized and semantically weighted. Consequently, SMS often assigns a less severe penalty for minor temporal misalignments compared to ARI's stricter, binary approach, leading to the expected higher scores observed in Figure 6.\n> \n> Regarding Figure 8, please note that the y-axis represents the error rate (calculated as 1 - score), where a lower value indicates better performance. Therefore, the observation that SMS is often lower than ARI in Figure 8 actually signifies that SMS assigns a *better* score, which is consistent with the trend shown in Figure 6. We will clarify the y-axis label in the final version to prevent any confusion.\n\n> **My major concern is that the proposed scores—especially SMS—are highly sensitive to the weighting scheme. While this flexibility is useful for application-specific evaluation, it becomes a downside when the metric is used for benchmarking. There should be at least some guidelines or constraints on how weights are selected. For example, do weights exhibit linear behavior—i.e., if \"delay\" is twice as important as \"missing segment,\" should its weight simply be doubled? Even with that assumption, the choice of weights still feels somewhat arbitrary. The paper would benefit from a sensitivity analysis or guidance on recommended weighting strategies for common use cases.**\n>\n> Thank you for raising this important concern. We acknowledge the trade-off between the flexibility of user-defined weights for application-specific diagnostics and the need for standardization in benchmarking.\n>\n> To answer your specific question: yes, the weights are designed to have a direct, linear impact on the penalty. The total penalty `P` is a weighted sum of individual error penalties. Therefore, if a \"delay\" error is considered twice as important as a \"missing segment\" error, its weight should indeed be doubled. This linear relationship is an intended feature, designed to make the tuning process intuitive.\n>\n> While the flexibility might seem arbitrary, the influence of these weights is formally bounded and predictable. The SMS score is fundamentally anchored by segmentation accuracy (`1 - E/N`, where `E` is the total error size and `N` is the series length), with the weights providing a controlled adjustment. Given SMS expression, the score is bounded by $1 - \\frac{(1+w_{\\max})E}{N} \\le \\text{SMS} \\le 1 - \\frac{E}{N}$, where $w_{\\max}$ is the maximum weight. Furthermore, the metric's sensitivity to changes in the weight configuration is directly proportional to the total error size `E`. Specifically, the absolute change in the SMS score for two weight sets, `w` and `w'`, is bounded by $\\lVert w - w' \\rVert_\\infty \\cdot (E/N)$. This property ensures that for reasonably accurate segmentations, the choice of weights has a limited and bounded impact, preventing arbitrary fluctuations. A detailed robustness analysis is provided in the appendix (p. 14).\n>\n> Regarding guidelines, we believe the best approach is to empower practitioners to make informed choices. SMS provides a detailed diagnostic breakdown, showing the size, type, and position of each error along with its associated penalty. This transparency allows users to observe the direct impact of their weighting scheme and tune it to reflect the priorities of their specific use case, moving the process from arbitrary to application-driven.\n\n> **To establish these metrics as state-of-the-art tools in the field, the paper would benefit from a broader set of experiments. While the selected datasets and models are solid, I recommend comparisons against basic metrics such as error rate or MSE, as well as sensitivity analyses for the proposed metrics. One particularly compelling experiment would be to show that using the proposed metric for model selection leads to better downstream performance. This would demonstrate that the metric not only captures meaningful differences but is also useful in practice—especially since much of the current analysis remains qualitative or relative to metrics with known limitations.**\n> \n> Thank you for these constructive suggestions, which help clarify the scope and utility of our work. We agree that comparing against basic metrics is important for contextualizing our contributions.\n> \n> Regarding error rate (or its complement, accuracy), this is implicitly included in our framework. As detailed in our response to your other concern about weight sensitivity, SMS simplifies to `1 - error_rate` when all weights are set to zero. This provides a clear and interpretable baseline for our metric.\nMean Squared Error (MSE) is not directly applicable to this task. The integer values of state labels are categorical and arbitrary; they do not carry numerical meaning that would make an MSE calculation valid. Even after the Hungarian alignment, the matched labels (e.g., `true_label=2`, `pred_label=3`) have no inherent numerical relationship. Converting the labels to a binary one-hot representation and computing MSE would be equivalent to calculating the point-wise error rate.\nRegarding the sensitivity analysis, we have addressed this in detail in our response to your other question about the weighting scheme and in the appendix (p. 14). We provide formal bounds on the metric's sensitivity to weight changes, ensuring that for reasonably accurate models, the score remains stable and predictable.\n> Regarding the suggestion to demonstrate improved downstream performance through model selection, our paper already provides the foundation for this. As shown in Figures 7 and 8, SMS offers a detailed error diagnostic that allows practitioners to identify the best-performing algorithm for a given dataset based on application-specific error priorities. This diagnostic can be used as-is for model selection. While a full experimental validation of the downstream impact is a key focus of our ongoing work, we suggest that current results already establish the practical utility of SMS for this purpose.\n\n> **Typo in algorithm: Line 8 in Algorithm 2 assigns the same weights to all components. I assume these are intended to be different weights, so this should be corrected.**\n>\n> Thank you for pointing out this ambiguity. Line 8 of Algorithm 2 uses the weight $w_e$, where the subscript $e$ denotes the error type identified in Line 7 (i.e., delay, isolation, transition, or missing). The notation is intended to select the specific weight corresponding to the given error type. We will revise the notations to make this more explicit and prevent any misinterpretation."}, {"review_id": "xDGmkWg7vw", "reviewer": "Reviewer_rSwh", "summary": "The paper introduces two novel measures the Weighted Adjusted Rand Index (WARI), enhances ARI by weighting segmentation errors based on their distance to true change points, improving sensitivity to temporal position; and the State Matching Score (SMS) a fine-grained measure that introduces a novel error typology (delay, isolation, transition, missing) and scores segmentation based on error type, position, and severity, allowing for customization and interpretability. These measures overcome the following limitations:\n1. Change point-based measures do not adequately capture the overall quality of the segmentation itself.\n2. Most widely used measures, such as Adjusted Rand Index (ARI), are point-based and thus treat all errors equally, failing to distinguish between different types of errors.\n3. Current evaluation measures cannot track and categorize the nature of errors (e.g., delay vs. isolation), leading to limited interpretability.\nThe paper empirically demonstrates that WARI and SMS outperform traditional measures in offering nuanced and actionable insights across synthetic and real-world datasets.", "questions": "1.\tThe typology assumes clear distinctions between isolation, transition, and missing. But these boundaries can blur under label noise or gradual transitions. Is this typology robust to subjective labeling or soft state transitions?\n2.\tSMS depends on penalty weights and boundary distance parameters. How sensitive are results to these choices? Essentially how does hyperparameter tuning affects evaluation?\n3.\tIf a segmentation algorithm is tuned with SMS in the loop, is it possible to \"overfit\" to SMS-friendly error distributions (e.g., producing short segments to minimize missing errors)?\n4.\tCan WARI be extended or adapted to domains outside time series (e.g., video segmentation)? i.e. is it generalizable?\n5.\tSMS involves Hungarian matching and error block scoring, potentially costly on long sequences or large batches. What are the real-world latency and memory footprints for high-frequency data streams?\n6.\tReal-world tasks often involve hierarchies or multi-scale segmentations (e.g., micro vs. macro activities). How do WARI and SMS handle hierarchical segmentations where ground truth may exist at different granularities?", "limitations": "Following are the limitations of the paper beyond author’s self-identified ones :-\n1.\tComputational Overhead and scalability concerns (In real time or high-resolution segmentation scenarios)\n2.\tParameter Sensitivity and Subjectivity\n3.\tLack of multi-scale support. The metrics assume a single-layer segmentation. They do not model segmentations where states can overlap or nest, which is common in real-world time series.\n4.\tWithout normalization for random agreement (as ARI and WARI do), SMS may reward trivial segmentations. A model predicting only one state could get a deceptively high SMS under certain weight configurations.\n5.\tThe paper evaluates segmentation in activity recognition and multivariate sensors. Generalizability to other domains remains untested.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper offers a thorough critique of ARI, F1 and covering scores.\n2.\tThe paper introduces a taxonomy of error types – delay, isolation, transition and missing – is a key conceptual contribution and add semantic depth, drawing a line between minor misalignment and severe structural failure.\n3.\tWARI is an elegant extension: it keeps the combinatorial structure of ARI, but adds a spatial awareness term, linking classification error with temporal topology. This weighting scheme can also be adapted to other point-based indices (as shown for NMI in appendix).\n4.\tSMS lets practitioners assign penalties to different error types. This makes the metric application-aware and useful in domain-specific scenarios.\n5.\tThe evaluation covers a wide array of datasets and segmentation algorithms, showing consistent benefits of WARI and SMS across both synthetic and real-world data.\n6.\tTraditional metrics like F1 or ARI depend on binary pair agreement. SMS instead scores over structured intervals, enabling continuous error gradients.\nWeakness:\n1.\tNo theoretical guarantees. While empirical evidence is strong, no formal guarantees are given on the properties of WARI/SMS (e.g., bounds, consistency, sensitivity). The measures are heuristically motivated, not axiomatized.\n2.\tSubjective Configuration of SMS. It depends on user-defined penalty weights for error types. Without a data-driven framework to derive these weights, scores can be arbitrary across studies.\n3.\tSMS is sensitive to class imbalance and vulnerable to inflated scores when the number of predicted segments is low. This is not true for ARI and WARI as they are chance-corrected measures.\n4.\tSegmentation algorithms could be tuned specifically to perform well on SMS (e.g., avoiding missing errors), possibly gaming the metric rather than improving true generalization.", "was_revised": true, "final_justification": "I have read authors' response to my comments. I have also considered their response to concerns raised by fellow reviewers. While some concerns still remain, I think the contribution is important, post clarification.", "rebuttal": "> **The typology assumes clear distinctions between isolation, transition, and missing. But these boundaries can blur under label noise or gradual transitions. Is this typology robust to subjective labeling or soft state transitions?** \n>\n> We acknowledge that ground truth labeling, typically performed by domain experts, can be subjective, especially in cases of gradual transitions. Our error typology is designed to be robust to such scenarios. Subjective or noisy boundary placements are generally captured as delay errors. As argued in the paper (Section 2.2.), these can often considered as less severe than major structural errors (e.g., missing or transition errors). Therefore, in most applications, users can assign a lower penalty to these delay errors, making the method less sensitive against subjective/gradual transitions. Conversely, in domains where precise boundaries are critical, one would expect highly accurate labels, and SMS allows for a correspondingly higher penalty on delays.\n\n> **SMS depends on penalty weights and boundary distance parameters. How sensitive are results to these choices? Essentially how does hyperparameter tuning affects evaluation?**\n>\n> Thank you for raising this concern. SMS indeed depends on user-defined parameters, providing some flexibility for application-specific evaluation. However, the boundary-distance terms are normalized and do not involve any user-defined hyperparameter, avoiding complexity and keeping the weights independent, so they do not interfere with one another, making the tuning more intuitive.\n> Moreover, the influence of the penalty weights on the final score is bounded and predictable. Formally, the SMS score is bounded by $1 - \\frac{(1+w_{\\max})E}{N} \\le \\text{SMS} \\le 1 - \\frac{E}{N}$, where $N$ is the time series length, $E$ is the total length of all erroneous segments, and $w_{\\max}$ is the maximum user-defined weight. This shows that the score is fundamentally anchored by the segmentation accuracy, with the weights providing a controlled adjustment. \n>Furthermore, we can demonstrate that the metric's sensitivity to changes in the weight configuration is directly proportional to the total error size $E$.\n> Indeed, the absolute change in the SMS score, for two sets of weights $w$ and $w'$, is bounded by $\\lVert w - w' \\rVert_\\infty \\cdot \\tfrac{E}{N}$, where $\\lVert w - w' \\rVert_\\infty$ is the maximum change across all weights. This property ensures that for a reasonably accurate segmentation (i.e., small $\\tfrac{E}{N}$), the choice of weights has a limited and bounded impact on the score, preventing arbitrary fluctuations and ensuring that the evaluation remains robust. This aspect has been verified experimentaly and detailed in the robustness analysis in the appendix (p. 14), showing that the impact of these parameters on the evaluation remains limited.\n\n> **If a segmentation algorithm is tuned with SMS in the loop, is it possible to \"overfit\" to SMS-friendly error distributions (e.g., producing short segments to minimize missing errors)?**\n>\n> This is a valid concern. While it is theoretically possible for an algorithm to adapt its behavior based on SMS penalties, it is important to note that the dominant component of the penalty for any error type is its size, as detailed above. Therefore, an algorithm cannot significantly improve its SMS score without fundamentally reducing the overall size of its errors. Our robustness analysis (Appendix, p. 14) demonstrates that the score is primarily driven by this factor, ensuring a strong baseline of evaluation.\n>\n> The user-defined weights for specific error types act as a secondary, fine-tuning mechanism. We view this not as \"overfitting\" but as a form of guided optimization, where the metric's flexibility allows practitioners to steer models toward application-specific priorities. For example, if avoiding \"missing\" errors is critical for a task, SMS can be configured to penalize them heavily, encouraging the algorithm to prioritize their reduction. The practical integration of SMS into the training loop for such targeted optimization is discussed as a key area for future work in Section 4.2.\n  \n> **Can WARI be extended or adapted to domains outside time series (e.g., video segmentation)? i.e. is it generalizable?**\n>\n>While our experiments focused on traditional time series data, the proposed metrics are designed to be generalizable. WARI and SMS operate on the temporal structure of segmentations—that is, a sequence of labels over time—and are agnostic to the underlying data modality.\n>\n>Consequently, for tasks involving temporal video segmentation, such as identifying scenes or actions, our metrics are directly applicable since a video can be treated as a sequence of frames (high dimensional multivariate time series). However, for spatial segmentation within individual video frames, which is not a one-dimensional temporal problem, these metrics would not apply. In essence, as long as the problem involves comparing a predicted temporal segmentation to a ground truth, our framework is suitable.\n\n> **SMS involves Hungarian matching and error block scoring, potentially costly on long sequences or large batches. What are the real-world latency and memory footprints for high-frequency data streams?**\n>\n> The computational complexity of SMS is determined by two main components: state alignment and error scoring. With `k_pred, k_true` the predicted and real number of clusters in the segmentation of a time series of length `T`:\n>\n> 1.  **State Alignment**: This involves building a `k_pred x k_true` cost matrix, which has a complexity of `O(k_pred * k_true * T)`, followed by running the Hungarian algorithm, which is `O(k³)` where `k = max(k_pred, k_true)`.\n> 2.  **Error Scoring**: This requires a single pass over the time series, resulting in a linear complexity of `O(T)`.\n>\n> The total complexity is `O(k_pred * k_true * T + k³)`. Since the number of states `k` is typically small and constant in most applications, the complexity is effectively linear with respect to the time series length `T`. The memory footprint is also linear, `O(T)`.\n>\n> In practice, the computational cost of SMS was negligible in our experiments compared to model training. Its linear scalability makes it efficient for offline analysis, even on long sequences or high-frequency data.\n\n> **Real-world tasks often involve hierarchies or multi-scale segmentations (e.g., micro vs. macro activities). How do WARI and SMS handle hierarchical segmentations where ground truth may exist at different granularities?**\n>\n> This is an interesting question that points to a more complex and distinct challenge in segmentation evaluation. Our current work focuses on \"flat\" or single-layer segmentation, where both the ground truth and the prediction exist at a single level of granularity.\n>\n> Consequently, WARI and SMS are not designed to handle hierarchical or multi-scale segmentations. Evaluating such structures would require a different conceptual framework capable of assessing agreement across multiple levels of abstraction simultaneously. This is a known limitation in the field, as highlighted in recent work such as the ClaSP paper (Ermshaus et al., 2023). To our knowledge, no established framework or real-world dataset currently exists for this specific task, with the exception of PaTSS (Carpentier et al., 2024), providing probabilitstic segmentation, but relying on synthetic dataset, and traditional measures for evaluation.\n>\n> We agree that developing metrics for hierarchical segmentation is an important direction for future research, but it is beyond the scope of the current paper."}]

Author rebuttal:

Meta-review: This paper introduces two novel evaluation measures, WARI (Weighted Adjusted Rand Index) and SMS (State Matching Score), to more accurately assess time series segmentation quality. WARI extends the traditional Adjusted Rand Index by weighting segmentation errors based on their temporal position, making it more sensitive to the timing of mismatches. The SMS metric provides a fine-grained analysis of errors, identifying and scoring four distinct error types: delay, isolation, transition, and missing segments. This approach offers detailed insights into error provenance and type, which is not possible with existing methods. The metrics allow for custom weighting of error types and have been empirically validated to demonstrate their diagnostic superiority.

The ARI is a widely used metric for clustering, so it is surprising that this paper successfully extends it to time series segmentation with WARI and SMS. Since robust evaluation metrics are crucial for developing better machine learning methods, this is likely a significant, long-term contribution to time series segmentation research. All the reviewers are happy about the paper, and I am also voting for acceptance.
Decision: Accept (poster)