Early Classification of Time Series: A Survey and Benchmark
Authors: Aurélien Renault
Alexis Bondu
Antoine Cornuéjols
Vincent Lemaire
Venue: Accepted by TMLR ()

Abstract: In many situations, the measurements of a studied phenomenon are provided sequentially, and the prediction of its class needs to be made as early as possible so as not to incur too high a time penalty, but not too early and risk paying the cost of misclassification. This problem has been particularly studied in the case of time series, and is known as Early Classification of Time Series (ECTS). Although it has been the subject of a growing body of literature, there is still a lack of a systematic, shared evaluation protocol to compare the relative merits of the various existing methods. In this paper, we highlight the two components of an ECTS system: \textit{decision} and \textit{prediction}, and focus on the approaches that separate them. This document begins by situating these methods within a principle-based taxonomy. It defines dimensions for organizing their evaluation and then reports the results of a very extensive set of experiments along these dimensions involving nine state-of-the-art ECTS algorithms. In addition, these and other experiments can be carried out using an open-source library in which most of the existing ECTS algorithms have been implemented (see https://github.com/ML-EDM/ml_edm).

Published in Transactions on Machine Learning Research (11/2025)
Early Classification of Time Series: A Survey and Benchmark
Aurélien Renault
rurelien.rennutt@8nnneeecom
Oronge Research AgroPanisTecdh
Alexis Bondu
aleris.bordduu@orregee.om
Orange Research
Antoine Cornuéjols AgroParisTech
Vincent Lemaire
SineenttlemaaireeSornngge...
Orange Research
Reviewed on OpenReview:
https: openreview. net/ net/forun? forum? id= bchDYmBicck
Abstract
In many situations, the measurements of studied phenomenon provided sequentially, and the prediction of its class needs to be made as carly possible so as not to incur too high time penalty, but not too early and risk paying the cost of misclassifcation. This problem has been particularly studied in the case of time series, and is known as Farly Classification of Time Series (ECTS). Although it has been the subject of a growing body of literature, there is still a lack of a systematic, shared evaluation protocol to compare the relative mcrits of the various existing methods. In this paper, we highlight the two components of ECTS system: decision and prediction, and foeus the approaches that separate them. This document begins by situating these methods within principle-based taxonomy. It defines dimensions for organizing their evaluation and then reports the results of a very extensive set of experiments along these dimensions involving nine state-of- the-art FCTS algorithms. In addition, these and other experiments can be carried out using an open-source library in which most of the existing ECTS algorithms have been implemented (see https://github. Com/ML-EDM/ml _dmm).
1 Introduction
In hospital emergency rooms (Mathukia et al. 2015), in the control rooms of national or international power grids (Dachraoui et al. 2015), in government councils assessing critical siluations, in many situations there is lime pressure to make early decisions. On the one hand, the longer decision is delayed, the lower the risk of making the wrong decision, as knowledge of the problem increases with time. On the other hand, late decisions are generally more costly, if only because early decisions allow one to be better prepared.
A number of applications thus involve making decisions that optimize trade-off between the accuracy of the prediction and its carliness. The problem is that favoring usually works against the other. Greater accuracy comes at the price of waiting for more data. Such compromise between the Earliness and the Accuracy of decisions has been particularly studied in the field of Early Classification of Time Series (FCTS), and introduced by Xing et al. (2008). ECTS consists in finding the optimal time to trigger the class prediction of an input time series observed over time.
As pointed out by Bondu et (2022). the ECTS problem rooted optimal stopping, of which is special case (Shepp) 1969; Ferguson 1989), where the decision to be made concerns both: (t) when to stop receiving
measurements in order to (i) predict the class of the incoming time series. To the best of our knowledge,
Alonso González & Diez (2004) are the earliest explicitly mentioning *classification when only part of the series are

---

Published in Transactions on Machine Learning Research (11/2025)
presented to the classifier". Since then, several researchers have continued their efforts in this direction and have
published a large number of research articles Xing et al.) 2009 Anderson et al.) 2012 Dachraoui et al. 2015 Mori
ct al. 2017a Schäfer & Leser 2020
Lv & Hu 2022 Ebihara et al., 2025
However, despite the growing interest in ECTS over the last twenty years (Gupta et al. 2020 Akasiadis et al., 2024 Santos & Kern 2016), there still remains a need for a shared taxonomy of approaches and an agreed well-
grounded evaluation methodology (sce Table 3 in Appendix [A.. Guidelines that we foel are important for a fair and informative comparison of existing ECTS approaches are listed below:
1. Cosls laken into accounl for evaluating the performance of the proposed method should be explicilly stated. It seems natural to distinguish between the misclassification costs, and the delay cost, and to add them in order to define the cost of making decision at time t. The delay cost may also depend on the true class y and the predicted one J, and a single cost function integrating misclassification and delay costs should then be used. For the sake of clarity, we keep the simple notation that distinguishes both cost functions in the rest of this paper.
2. The performance of the proposed methods should be evaluated against a range of possible types of cost functions. It is usual to evaluate "by default" the methods using 4--1 loss function that penalizes wrong classification by a unity cost, and to consider linear delay cost function. However, lots of applications rather involve unbalanced misclassification costs, and possibly also non-linear delay costs.
3. The contributions of the vorious components of an ECTS algorithm should be as clearly delineated as possible. The predominant approach to ECTS is to have a decision component which is in charge of evaluating the best moment to make the prediction about the class of the incoming time scries, and a classifier one which makes the prediction itself. We call these methods "separable methods". In order to fairly compare the triggering methods at play, which are at the heart of ECTS, the classifier used should be the same for all methods in the experiments.
Recent approaches mix the decision and the prediction components. This is the case of using neural networks. These methods do not allow the evaluation of the merits of the decision component by itself, independently of the classification component, and they are accordingly not considered here.
methods
4. Performance obtained should be compared with that of "baseline" algorithms. Failing to do this weakens any claim about the value of the proposed method. In the case of ECTS tasks, two naive baselines are: (1) make a prediction soon it is allowed, and (2) make prediction after the entire time series has been observed. In our experiments reported in Section 4 we have added third baseline less simple than the two aforementioned, which, to our knowledge has never been published as an original method. This is a confidence--bssed meth where a decision is triggered as soon as the confidence for the likeliest class given X₁ is greater than a threshold.
5. The dalasels used for truining and lesling should remove the biases that are specific to the ECTS task and that may result in erroneous evaluations. A case in point, concerns the normalization often used in time series datasets. Dau et al. (2019) have reported that 71% of the reference time series classification datascts used to evaluate ECTS methods are made up of z-normalized time series, Not only, this setting is not applicable in practice, as z-normalization would require knowledge of the entire incoming time series, but it could also introduce a leakage of information from the future of the time series to the previous time steps which is detrimental to fair assessment of the methods (Wu et al. 2021
In this article, we aim to present a methodology for a fair and informative comparison between separable ECTS methods in the literature. Specifically, this paper makes the following contributions:
A taxonomy is proposed in Section 2 classifying separable approaches in the literature according to their design choices.
Extensive experiments have been performed, taking into account the obstacles mentioned above. (1) The experimental protocol in Section 4. explicitly defines the costs used during training and evaluation, and varies the balance between misclassification and delay costs by using a large range of cost values.
2

---

Published in Transactions on Machine Learning Research (11/2025)
(2) Experiments are performed repeatedly for several types of cost function, i.c. balanced or unbalanced misclassífication costs, and linear or exponential delay costs (see Sections 4.2 and 4.3) and many intermediate results are available in the supplementary materials. (3) Ablation and substitution studics are conducted in Section 4.4 with the aim of evaluating the impact of methodological choices, such as the choice of classifier, its calibration, or even z-normalization of training time series, as well as the non-myopia property of some trigger functions. (4) The experiments include three baseline approaches, rarely considered in the literature, which often prove to be efficient. (5) In addition to the reference data used in the ECTS field, collection of some thirty non-z-normalized datasets is proposed and provided to the community.
An open source library is made available to enable reproducible experiments, as well as to facilitate the scientific community's development of future approaches. Particular care has been taken to ensure the quality of the code, so that this library may be used to develop real-life applications (sce Appendix D
The scope of this paper is survey and benchmark of separable ECTS approaches, in line with our guideline #3. Interested readers can refer to Appendix C for complementary results on end-to-end recent approaches. The rest of this paper is organized as follows: Section 2 proposes and describes an ECTS taxonomy in order to situate separable approaches in relation to each other. It underlines the different choices to be made in a well-founded way when designing an ECTS method. Section 3 presents a review of state-of-the-an methods for ECTS organized along the dimensions of the suggested taxonomy. In Section 4 we present the pipeline developed to conduct extensive experimentation, and we report the main results obtained for different cost settings. Finally, Section 5 concludes this paper.
Position with respect to other literature surveys In reaction to the abundant scientific activity around ECTS, Gupta et al. (202)) wrote survey, however with the following characteristics. First, their perspective on ECTS is confidence-basee based as they declare "A primary task of an carly classification approach is to classify an incomplete time series as soon as possible with some desired level of accuracy" (italics are ours) and it is not centered on optimizing the tradeoff between predictive accuracy and earliness. Second, they organize the covered approaches by application domains and methods for representing time series, viz. prefix- based, shapelet- based, mrodel-based, and miscellancous. A perspective that, in a way, tangential to the problem of selecting the right decision time.
Another paper Akaasiadis et al., 2024) intends to propose a framework to evaluate ECTS systems. Noticeably, it also adopts comfidence-based approach: "The objective is to find the earliest time-point of time-series at which a reliable prediction can be made". Alongside the survey, a benchmark is proposed whose characteristics are detailed in Table 3
The earliest survey published in 2016 by Santos & Kern 2016 deals mostly with classification methods for complete time series. The final section describes four ECTS algorithus with no experiments.
2 ECTS: concepts and taxonomy
The aim of this section is (f) to formally describe the ECTS problem, and (it) to outline in a principled way the various choices that need to be made when designing one ECTS method.
Problem statement
In the ECTS problem, an input time series of size T is progressively observed over time. At time t S T, the incomplete time series = (fi, is available where Tigagrs)) denotes the time-indexed measurements. These measurements can be single or multi-valued, The input time series belongs to an unknown class E y. The task is to make a prediction 9 E y about the class of the incoming time series, at a time f E (1,7) before the deadline T.
An ECTS approach aims at optimizing a trade-off between accuracy and earliness of the prediction, and thus must be evaluated on this ground. The correctness of the prediction is measured by the misclassification cost Cm(0ly) where ý is the prediction and y is the true class. The time pressure is sanctioned by delay cost Ct(() that is assumed to be positive and, in most applications, an increasing function of time. We thus consider:
1 (see httpa://github Com/ML-Eae EDMM/DEnndnnnnn
3

---

Published in Transactions on Machine Learning Research (11/2025)
Cm (0ly): : 3 y + R, that corresponds to the cost of predicting 9 when the true class is y.
Ca(t): R+ + R, the delay cost that, usually, is a non-dccreasing function over time.
An ECTS function involves a predictor f(x),, which predicts the class of an input time series x, for any E [1, 7). The cost incurred when a prediction has been triggered at time given by a loss function ((D(x)).y,0) = Cm (f(x)) y)) Ca(t). The best decision time 1* is given by:
arg min min (()(x)), y,
(1)
rE[1.T]
Let s* E S an optimal ECTS function belonging to class of functions S, whose output at time & when receiving X is:
if extra measures are queried; *xx,) = { 0 10" - g(xx) when prediction is triggered at t= t";
(2)
ECTS is however an online optimization problem, where at each time step t a function s(x;) must decide whether to make a prediction or not. Equation | is thus no longer operational since it requires complete knowlodge of the time series. In practice, the function s(x,) triggers a decision at i, based on a partial description X₃ of the incoming time series xy (with 8 < T). The goal of an ECTS system is choose a triggering time ł as close as possible to the optimal one t", at least in terms of cost, minimizing L(((x;), 8) ((0(x)), ),,,*) as much possible.
From machine learning point of view, the goal is to find function E S that best optimizes the loss function C, minimizing the true risk over all time series distributed according to the distribution P (xxx) that governs the time series in the application:
mrgmin E(x.u))P (c.
(3)
In order to solve the ECTS problem, a training set composed of M labeled time series, denoted by (x², y jEjo E (X's x y), where each series Xr = (aI,..., Rr) is complete and of the same size T, and associated with its label y E Vi is used to learn how to predict the class of an incoming time series x = (T)) and to learn what can be expected in the future given X₃.
Consequently, model training and deployment are of different natures. The training stage is carried out supervised batch process, with access to the full labeled time series. When it comes to testing, on the other hand, decision- triggering is an online process which stops at time I, and at the latest, when the deadline T is reached.
2.1 Separable vs. end-to-end approaches
An ECTS function must solve both the question of (i) when to slop receiving new measurements and decide to make a prediction and (ii) how to make the prediction about the class of the incoming time series X
In the separable approach, these questions are solved using two separate components. The classification one deals with making a prediction: x, + û, while the trigger function decides when to predict. Within this perspective, the classification component is learned independently of the trigger one, while the latter uses the results of the classification component in order to trigger decision. We formalize scparable approaches by:
s(x,) = (g (x,)
(4)
where g is the decision trigger function, and h is the prediction function.
In the end-to-end approaches, single component decides when to make prediction and what that prediction is.
Thus, the function 8, defined in Equation 2 is responsible both for choosing the time f. for making the predictions, and for the prediction itself g.
Notice that the notation X' is abuse that use use to simplify our purpose. In all mathematical rigor, the measurements observed successively constitute a family of time-indexed random variables (x.) This stochastic process x is not generated commonly by distribution, but by filtration F = (Fi)e(().)) which is defined as a collection of nested d-algebras (Klenke) 2013) allowing to consider time dependencies. Therefore, the distribution X should also be re-written filtration.

---

Published in Transactions on Machine Learning Research (11/2025)
The question that naturally arises is which type of architecture (i.e. end-to-end or separable) performs best. On the one hand, in separable approaches, the classification component trained independently of the triggering one, which can be detrimental, for example, by propagating errors from one module to another. On the other hand, separating the ECTS problem into two inherently simpler sub-problerns could be an advantage, for example; terms of convergence during training. Additionally, the separable framework allows one directly leverage the latest advances in the Time Series Classification (TSC) literature (Bagnall et al. 2017 In this paper, do not delve any further into the question of which type of architecture is best, which leave for future work.
The rest of this section is specific to separable ECTS approaches, which represent a large part of the literature. In the following, we first examine the decision component (i.e. the trigger function) and then the prediction one (i.c. the classifier). Figure 1 gives a general view of the proposed taxonomy.
2.2 Trigger function's properties
The trigger function is responsible for finding when to make prediction. It triggers the prediction at time & such that [(j(x;)) f) is close as possible to the optimal cost C(y" (xe)), y,t for the optimal decision time t*.
Separable ECTS architecture
g
h Classification function
Trigger function
Properties:
Cost informed testing time
Single classifier
Confidence based
Or Classifier collection
instant-basee based sequence-based based
Anticipation based
Proposed taxonomy for separable ECTS approaches, which can be viewed a function composition of trigger and classification functions. The taxonomy does not follow tree structure the trigger function may have several properties, that not mutually exclusive. For instance, trigger function can be both confidence-based and anticipation based.
Figure 1:
2.2.1 Cost-informed or cost-uniformed
The first property is whether or not the trigger function actually takes as input any cost functions (or eventually some proxy of them) during training. We call these approaches cost-informed. On the contrary, cost-uninformed approaches are usually based on some hard pre-defined rules trigger and are not always easily expandable to generic cost- setting framework. Additionally, when being cost-informed, trigger functions leverage cost information even more, by being adaptable at testing time, i.e. during inference, trigger decisions directly depend on cost functions.

---

Published in Transactions on Machine Learning Research (11/2025)
2.2.2 Confidence-basse approaches
The simplest trigger model of this kind consists in monitoring a quantity related to the confidence of the prediction over time and triggering class prediction as soon as a threshold value exceeded. The confidence metric monitored can take different forms. For example, a baseline approach, referred to as Probo. Threshole in the remainder of this paper, involves monitoring maxyey p(y|>,, the highest conditional probability estimated by the classifier. This baseline example is qualified as instont- basen method, since it takes input only the last confidence score available at time& t. Another type of approaches, qualified as sequence-based, monitors the entire sequence of past confidence scores, and a triggering prediction is made conditionally on a particular property of this sequence. Accordingly, trigger functions can either take as imput a scalar value, e.g. ((mayye> p(y]x))), in the case of instant-based approaches, or a sequence of scalar values, e.g. g( g(mmax ( (ssss,,, in the case of sequence-bosed approaches (sce Section 3.1.
Those kind of examples can be described myopic since they only look at the current time step t (or past ones well in the sequence- based case), without trying to anticipate the likely future, i.e. not directly having decision made based on a forecast of some form. The anticipation-bost based approaches do this.
2.2.3 Anticipation-based decisions
As was first noted by Achenchabe et al. 20021; , the ECTS problem can be cast as a LUPI (Learning Using Privileged Information) problem (Vapnik & Vashist 2009). In this scenario, the learner can benefit at the training time from privileged information that will not be available at test time. Formally, the training set can be expressed as T= {(x,, ()),, where x, is what is observable and x is some additional information not available when the prediction must be made. This exactly what. happens the ECTS problem. Whereas at test. time, only x, is available, during training the complete time series is known. This brings the possibility to learn what are the likely futures of an incoming time series x, provided it comes from the same distribution. Hence, it becomes also possible to guess the cost to be optimized for all future lime steps, and therefore to wail. until the moment seems the best. This type of approach can be said anticipation-busse -assd (also called non-myopic in the literature). They can come with many flavors, as long as their decisions are based on some kind of anticipation (see Section 3.2). Because more information from the training set is exploited, it can be expected that these methods outperform myopic ones.
Is this confirmed by experience? Are there situations where the advantage significant? Our experiments Scction f provide answers to these questions.
2.3 Choice of the classification component
The role of the classification component is to return the prediction 9 of the class of the incoming time series X; at the time decided by the trigger function: 9 = h)(xx).
One source of difficulty when devising an ECTS method in the separable setting is that, at testing time, inputs differ from one time step to another. When an incoming time series is progressively observed, the number of measurements, and hence the input dimension, varies. Two approaches have been used to deal with the problem.
1. A set of classifiers (hehep.. is learned, each dedicated to a given time step t, and thus a given input dimension. In practice, authors often choose limited subset of timestamps, usually a set of twenty (one measurement every 5% of the length of the time series), to restrict the number of classifiers to learn and therefore the associated computational cost.
2. A single classifier h is used for all possible incoming time scries x,. One way of doing this is to "project" an input x, of dimension L x d, if d is the dimension of an observation at time / (i.e. multi-valued time series), into a fixed dimensional vector whatever t and d. This may simply be the mean value and standard deviation of the available measurements (multiplied by the dimension d) or the result of a more sophisticated feature engineering as tested by [Sakkun et al. (2017). Deep learning architectures can also be used to learn an encoding of the time series in an intermediate layer (Wang et al. 2016 Sawada et, al. 2022
Baseline implemented in the acon Middlehurst et al. 2024a library:
https://urlz.fs fr/q/l/

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Table of published separable methods for the ECTS problem with their properties along dimensions under- lined in the laxonomy. Note that void values indicate that the corresponding properly is not present in the referred system.
References
Classifier(s) (collection v
Confidence
Anticipation Cost informed
Reject Hatamm & Chira.
2013
SVM
instant
iHMM Antonucci al.
2015
HMM
instant
ECDIRE Mori et 2017b 7b
Gaussian Process
instant
Stopping Rule Mori et al.
2017a
Gaussian Process
instant
ECEC
Lv et al.
2019
WEASEL
sequence
TEASER Schäfer & Loser
2020
WEASEL (V)
sequence
SOCN Lv et al.
2023
FCN
sequence
ECTS
Xing et al.
2012
INN
instant
ReIClass Parrish al.
2013
QDA, Linear SVM
instant
2step/NoCluster Tavenard Malinowski
2016
Linear SVM (V)
(test) (test)
ECONOMY - mmax Zafar et al.
2021
XGBoost + tsfel (V)
CALIMERA Bilski & Jastrzebska
2023
MiniROCKET (V)
FIRMBOUND Ebihara al.
2025
LSTM
Both approaches have their own limitations. On the one hand, using a set of classifiers, each independently dedicated to a time step, does not exploit information sharing. On the other hand, using a single classifier seems to a more difficult task, as the representation of x, can be different at times and t + and all further time steps which can lead to additional difficulty for the classifier while moreover requiring more demanding feature engineering step. Therefore, here also, it is interesting to mcasure experimentally whether one dominates the other. This will be the subject of future work.
3 An organized state-of-the-art of separable methods
In this section, approaches from the literature are considered and organized around two key notions from the intro- duced taxonomy: confidence-bused and anticipation-bused.
Subsection 3.1 presents methods whose decisions are triggered in a myopic way, based on some confidence measure.
Subsection 3.2 describes approaches using a non-myopic decision criterion, which attempts to anticipate likely contin-
uations. Table shows how various approaches described the literature can be organized using the characteristies underlined in the proposed taxonomy.
3.1 Confidence-based approaches
There exist two families of confidennc-bassed approaches. In the first one, only the last time step is considered, a score based on confidence estimations is monitored at each time step and class prediction is triggered soon as a threshold on this score is exceeded. By contrast, in the second, a sequence of estimated scores is monitored, and the condition to trigger decision depends upon property of this sequence.
3.1.1 Instant-based decision criterion
One basic method to monitor maxgey p());)), the highest conditional probability estimated by the classifier, which is a simple measure of classifier confidence over time. As soon it exceeds a value, which is hyperparameter of the method, a prediction is made. We call this method PROBA THRESHOLD and use it baseline for comparison later in our experiments.

---

Published in Transactions on Machine Learning Research (11/2025)
The REJECT method (Hatami & Chira 2013) uses ensemble consensus as a confidence measure. For each time step, first (f), a pool of classifiers is trained by varying their hyperparameters (i.e. SVMs); then (4t), the most accurate of these are selocted; and (iit) the pair of classifiers minimizing their agroement in prodictions is chosen to form the ensemble. Finally, the prediction is triggered as soon as both classifiers in the ensemble predict the same class value.
Hidden Markov Models (HMMs) are naturally suited to the classification of online sequences. An HMM is learned for each class, and at each time step t, the class to be preferred is the one with the highest a posteriori probability given X,. However, the decision to make a prediction now or to postpone it must then involve a threshold so that the prediction is only made if the a posteriori probability of the best HMM is sufficiently high or is greater than that of the second-best. In reaction to this, Antonucei et al. (2015) propose to replace the standard HMM with imprecise HMMs based on the concept of credal classification. This eliminates the need to choose threshold, since a decision is made when one classification "dominates" (according to a critcrion based on probability intervals) all the others.
Rather than considering only the largest value predicted by the classifier, it is appealing to consider also the difference with the second largest value, since a large difference points to the fact that there is no tie between predictions to expect.
This is one dimension used in the Stopping Rule (SR) approach (Mori al. 2017ab. Specifically, the output of the system is defined as:
s(((x))) = 0
if extra measures are queried;
(5)
( arg yay p(y)x)) when 7n1 P1 + 12 Pa + 7 7 >
where p1 is the largest posterior probability p(v)x;) estimated by the classifier h, P2 is the difference between the two largest posterior probabilities, and & represents the proportion of the incoming time series at time 4. The parameters Y1, 72, and 7/ are learned from the training set, using genetic algorithm.
Using the same notations as SR, the Early Classification framework based on class Discriminativeness and RF- Liability (ECDIrE) (Mori et al. 2017b) finds the earliest timestamp for which threshold applied on pl is reached (defined as in Equation 5) Then, the quantity P2 is monitored, and a second threshold is applied to trigger the prediction.
Ringel et al. (2024) use the Learning Then Test (LTT) (Angelopoulos et al., 2021 calibration framework to address ECTS. In practice, the proposed approach greedily computes thresholds at each time step, in order to monitor some conditional control risk measure, given pre-defined error rate. This paper investigates text applications.
Historically, a related scenario predates the ECTS problem but is different, In the sequential decision making and optimal statistioal decisions frameworks (DeGrool. 2005 Berger 1985), the successive measurements are supposed be independently and identically distributed (i.id.)) according to a distribution of unknown "parameter" 6. The problem is to determine as soon as possible whether the measurements have been generated by a distribution of parameter 0g (hypothesis Ho) or of parameter O, (hypothesis Hi) with 0p + 0,. In the Wald's Sequential Probability
P((a).....))) ==-1) Ratio Test Wald & Wolfowitz. 1948 Ghosh & Scn 1991), the log- likelihoddood ratio R₂ = log P( y=+1) is computed and compared with two thresholds that are set according to the required error of the first kind a (false positive error) and error of the second kind B (folse negative error). This beautiful setting allows one to get optimal decision times at the cost of being able to compute the ligelioil However, it differs from the ECTS problem, where successive observations are dependent. The i.i.d. assumption being not valid for the ECTS problem, a generalization to the non-i.i.d. case was proposed by [Tarakkvssy et al. (2014), providing guarantees for the asymptotic case (with T - oo). Despile this latter limitation, Ebihara et al. (2025 has recently applied this lype of approach to ECTS with finite time horizons. The authors propose practical ways of both estimating Rg (Ebihara ct al. 2023) and triggering times by solving a backward induction problem (Tataakossyy et al. 2014).
3.1.2 Sequence-basse based decision criterion
Other approaches propose sequence- based confidence measures specifically designed for the ECTS problem.
The Effective Confidence-based Early Classification (ECEC) (Lv et al. 2019 proposes confidence measure based on the sequence of predicted class values, from the first one observed to the current timestamp. At each time step,
8

---

Published in Transactions on Machine Learning Research (11/2025)
this approach exploits the precision of the classifier to estimate the probability for each possible class value y E y of being correct if predicted. Then, assuming that successive class predictions are independent, the proposed confidence measure reprosents the probability that the last class prediction is correct given the sequence of predicted class values. The proposed confidence measure is monitored over time, and prediction is triggered if this measure exceeds a certain threshold y tuned as the single hyperparameter.
The TEASER (Two-tier Early and Accurate Series classifERR) (Schäfer & Leser 2020) approach considers the problem of whether or not a prediction should be triggered as a classification task, the aim of which is to discriminate between correct and bad class predictions. As the authors point out, the balance of this classification task varies according to the time step considered E [1. Indeed, assuming there is an information gain over time, there are fewer and fewer bad decisions as new measurements are received (or even no bad decisions after a while, i.e. V t > t' (0 < t' I T) for some datascts). To exploit this idea, a collection of one-class SVMs is used, learning hyper-spheres around the correct predictions for each time step. A prediction is triggered when it falls within these hyper-spheres for v consecutive time steps.
The Second- Order Confidence Network approach (Socx) (Lv et al. 2023 considers, as does TEASER, the same classification task aiming to discriminate between correct and bad predictions. To learn this task, transformer (Vaswani et al., 2017) is used, taking input the complete sequence of conditional probabilities estimated by the classifier h, from the first time step, up to the current time step. A confidence threshold is learned by minimizing the same cost function as Lv et al. (019 do, above which the prediction is considered reliable and therefore triggered.
3.2 Anticipation-basee methods
One way of designing approaches that anticipate future mcasurements to achieve classification of an incomplete time series while guaranteeing minimum probability threshold, according to which the same decision would be made on the complete series. This is the case of the Reliability Classification (REI.CLASS) approach (Parrish et al., 2013). Assuming that the measurements are i.idd and generated by Gaussian process, this approach estimates p(xr x1) the conditional probability of the entire time series xy given an incomplete realization x, and thus derives guarantees of the form:
((r(x))=))x) /
p(XT xx) dxt 2 7
XT (r(())
where XT is a random variable associated with the complete time series, y is a confidence threshold, and br is the classifier learned over the complete time series. At each time step ((T( = y y/x,) is evaluated and a prediction is triggered if this term becomes greater than the threshold y, which is the only hyperparameter to be tuned.
Another way of implementing anticipation-based approaches is to exploit the continuations of training time series, which are full-length. One of the first methods for ECTS has been derived into such an anticipation-bo based approach. The first, called Early Classification on Time Scries (EcTs) (Xing al., 2009), exploits the concept of Minimum Prediction Length (MPL), defined as the earliest time step for which the predicted label should not change for the incoming time scries x; from t to T. This is estimated by looking for the INN of x, in the training sct, and checks whether from l onward, its predicted label does not change. To be more robust, the MPL is defined based on clusters computed on full-length training time series to estimate the best decision time. The approach has been extended later on to speed up the learning stage (Xing et al.] 2012). This method looks its own way at the likely future of X, i.c. an incomplete time scries belongs to a cluster whose continuations are known and thus can be considered an anticipationn-base method.
Dachraoui et al. 2015)) present a method that claims explicitly to be "non-myopic" in that a decision is laken at time t only insofar as it seems that no better time for prediction is to be expected in the future. In order to do this, the family of ECONOMY mothods estimates the future cost expectation based on the incoming time scries x;- This can be done since the training data consists of full-length time series and therefore a Learning Using Privileged Information (LUPI) (Vapnik & Vashist 2009) is possible.
More formally, the objective is to trigger a decision when Ey.o[C(). U, [x] is minimal, with:
T..(C((.vt)) t)/x,] = Σ P(y]x,)
Σ
P(oly, x.) C. (0lv) + Calt)
(6)
wEY
De
9

---

Published in Transactions on Machine Learning Research (11/2025)
A tractable version of Equation 6 has been proposed by introducing an additional random variable which is the membership of x, to the groups of a partition g:
E P(en(x)) (x,)
Σ NEV
P(ylax) Σ BEy
94 Ca(tt
(7)
DREG
In technical terms, training approaches from the ECONOMY framework involve estimating the three probability terms of Equation for the current time step t, as well as for future time steps t + E [t 1,,, with:
(x)) the probability of x, belonging to the groups gh g
P(y (x) the prior probability of classes in each group,
P(g y. 9h) the probability of predicting 9 when the true class y within the group gk-
A key challenge in this framework is to design approaches achieving the most useful partition for predicting decision costs expectation. In the first article which presents this framework Dachraoui et al. (2015). a method, called ECONOMY-K, is designed as follows. (i) A partition of training examples is first performed by a K-mcans algorithm (ii) then a simple model uses the Euclidean distance proxy of the probability that x, belongs to each group; (fi) the continuation of training time series within each group is exploited to predict the cost expectation for future time steps.
In order to avoid the clustering step with the associated choice of hyperparameters Tavenard & Malinowski 2016) presented a variant called NOCLUSTER which uses the 1-ncarest neighbor in the training set in order to guess the likely future x,.
Then, ECONOMY-r was introduced by Achenchabe et al. (2021) which relies on a supervised method to define a confidence-based partition of training time scries. The algorithm, dedicated to binary classífication problems, is designed as follows: (i) a collection of partitions is constructed by discretizing the output of each classifier (he)e into equal-frequency intervals, the groups thus formed correspond to confidence levels for each time step; (ii) at the current time t, the incoming time series belongs to only one group, since the output of the classifier he falls within a particular confidence level; (iiz) then, a Markov chain model is trained to estimate the probabilitics of the future time step confidence levels. ECONOMY-7-MAX (Zalar el al., 2021] generalizes this approach to multi-class problems, aggregating the multiple conditional probabilities in the classifiers' output by using only the most probable class value.
CALIMERA (Bilski & Jastrzebska, 2023 uses anticipation about the future from another perspective. Instead of trying to guess the likely continuation of x, which allows one to compute expected future costs, and therefore to wait until there seems no better time to make prediction, their method is based on predicting directly the difference in cost between predicting the class now and the best reachable cost from next timestep to last one T. If this difference is positive, then it is better to postpone the prediction. They advocate furthermore, that a calibration step should intervene on the collection of classifiers, in order to build meaningful regression target to learn the trigger model.
4 Experiments & Results
This section presents the extensive set of experiments carried out in order to provide a consistent and fair evaluation of wide range of existing litcrature's methods. We first describe the experimental protocol used. We then turn to the experiments and their results. Figure provides synthetic view of the organization of these experiments.
Section 4.1 introduces the experimental protocol as well as the global evaluation methodologies.
In Section 4.2 eight state-of-the-art methods and the three baseline ones evaluated using widely used cost setting, T.e. with binary balanced misclassification cost and a linear delay cost.
10

---

Published in Transactions on Machine Learning Research (11/2025)
In Section 4.3 methods are tested in an anomaly detection scenaric where the misclassffication cost matrix is severely imbalanced, with false negatives being much more costly than false positives, and where the delay cost is no longer linear with time but increases exponentially with time.
Finally, Section briefly describes set of other experimental setups, derived from either the standard setting or the anomaly detection one, including, for instance, testing the impact of z-normalization. Com- plementary results can be found in Appendix
Standard cost setting
Ca linear Cn balanced
4.3
Anomaly detection cost. setting
Cj exponential Cm imbalanced
Legendary of datasets:
4.4
Conplementary results
Collection E.1
(TT)
mostly z-normalized
Ablation/sabesiitttinn studies
Collection E.2 (34).
Derived from
non z-normalized
Removing calibration (Appendix
F.3
Collection E. z-normalized
(34).
Removing non-myopia (Appendix
Impact of base classifier (Appendix
F.5
Collections
imbalanced versions
Impact of z-normalization (Appendix
F.6
Derived from
Exponential delay cost only (Appendix F.7
Imbalanced cost. only (Appendix F.7
Standard cost setting (Appendix
F.7
Figure 2: Experiments diagram. While this section mainly discusses results about the cost settings in Subsections
4.2 and 4.3
many other alternative experiments are briefly analyzed in Subsection 4.4 and are more detailed in the Details on the used datasets can be found in Appendix
A Appendix ix
The experiments presented here aim to evaluate the effects of design choices on method performance and thus provide answers to the questions:
Do anticipation--uused methods perform better than myopic ones?
Do methods that are cost-informed for their decisions (ie. explicitly estimating costs) perform better than methods that are cost-uninformed? (sec Section 2.2.1
How do the various methods fare when modifying the form of the delay cost and/or the misclassification cost matrix?
"We consider anomalies to be actual phenomena of interest, such the failure of a machine (oniol al. 2024 These anomalies be accompanied by revealing precursor signals in the time series, which ECTS system should be able detect by optimizing both the accuracy of prediction and its carliness.

---

Published in Transactions on Machine Learning Research (11/2025)
4.1
Experimental Protocol
This section covers the shared part of the experimental protocol for all the experiments, irrespective of the choice of the cost functions (see Sections 4.2 and 4.3 for this). Additional implementation specifications can be found in Appendix B
4.1.1 Evaluation of the performance
For each test. time series x', an FCTS method incurs a cost assumed to be of the additive form: Cm (flli)) + Ca((), where f the time when the system decided to make a prediction, this prediction being d:-
For a test set of M time series, the average cost of method (Equation g is used as the criterion with which evaluate the methods.
AugCostrest
Σ Cmille)) Ca(fi) M
1
(8)
In addition, in order assess how the methods adapt to various balances between the misclassificatiin and the delay costs, we vary the settings of these costs by weighting them during training and testing, The performance of the methods is therefore evaluated using the weighted average cost, defined in Equation for different values of the costs balance a, ranging from 0 to 1, with a 0.1 step:
AugCosta 1
1 E ax Cm + aa) Ca(li) M
Small values of correspond to high delay cost and a small misclassification cost ; inversely, large values of give more weight to the misclassification cost with lower delay cost.
4.1.2 Comparing the trigger methods
Our experiments aim foremost comparing the trigger methods used that are responsible for deciding when to make prediction about the class of the incoming time series. To this end, all compared methods in our experiments use the same prediction component. As advocated by Bilski & Jastrzebska (2023), we have chosen the MINIROCKET algorithm (Dempster et al.] 2021) to be the base classifier for all methods. It is indeed recognized as among the best performing classifiers in the time series classification literature as well as one of the fastest ones. However, we have carried out additional experiments with two other classifiers (see Section 4.4p. our experiments, we have additionally reported results for end-to-end methods in order to situate the performance of these methods respect separable ones Appondix C.2
Trigger models: Eight. trigger models selected from the literature based their usage and their performances
ECONOMY-y-MM. MAX AAchnchabe et al. 2021), CALIMERA
Bilski & Jastrzebskal 2023). STOPPING RULE (Mor et al.
2017a), TEASER (Schäfer & Leser 2020 TEASER
ECEC
(Lv et al. 2019 ECDIRE Mori et al. 2017b). ECTS
He et al. 2013).
All these methods have been re-implemented using Python, reproducing results close to the published ones. Except for the code for the ECTs implementation, which has been taken from Kladis et al.] (2021). Hyperparameters are the ones chosen in the original published methods. Code to reproduce the experiments is available publicly at (see https://github.. http://gittuuu..coMMMLL---
Baselines: Furthermore, in order to evaluate the benefits, if any, of the various methods, it is telitale to compare them with simple ones. We chose three such baselines:
ASAP (As Soon As Possible) always triggers prediction at the first possible timestep.
5The EDSC algorithm Xing al. 2011), even though available in the provided library, is not included in the following experiments, due high space and time complexity (which hinders fair comparisons.)
12

---

Published in Transactions on Machine Learning Research (11/2025)
ALAP (As Late As Possible) always waits for the complete series to trigger the prediction.
PRODA THRESHOLDD is a natural, confidence-based, cost-informed, baseline: it triggers a prediction if the estimated probability of the likeliest prediction exceeds some threshold, found by grid search (cf. Section manfideene-based).
3.1
4.1.3 Datasets and training protocol
Two collections of datasets In order be able to directly compare our results to past experiments, we first use the usual TSC datasets from the UCR Archive (Dau al., 2019) with the default split. In total, we have used 77 datasets from the UCR Archive, i.e. the ones with enough training samples to satisfy our experimental protocol from the start to the end, i.c. with at least one example per class within cach of the used disjoint subsets (sce Splitting
strategy below). (blue cylinder in Figure 2). In this way, most of the datasets used by either Mori el al. (201 and Lv et al. 2019 or by Achenchabe et al. 2021 are contained in our experiments.
A second collection of non z-normalized datasets is also provided. In this way, the associated potential information leakage is avoided (see Section []. Any difference in the performance obtained on the z-normalized data sets can thus signal the danger of z-normalization with firm evidence. Considering the limited amount of non z-normalized datasets within the UCR archive (Dau et al., 2019), we have decided to look for complementary new datasets so as to provide another collection of datasets. To this end, the Monash archive for extrinsic regression (Tan et al.. 2020), provided 20 new time series datasets, for which have discretized the numeric target variable into binary classes based on a threshold value. For instance, if this threshold is equal to the median value of the regression target, the resulting classification datasets will be balanced in terms of classes (as in Section 4.2). Note that this threshold can be chosen differently to get imbalanced datasets (as in Section 4.3.2), several thresholds could also be used to increase the number of classes. As a result, we obtaín new set of classification tasks, as has recently been done by Middlehurst al. (2024b). In the end, 34 datasets have been gathered: 14 from the original archive and 20 from the Monash extrinsic regression archive. (orange cylinder in Figure 2b. More details about the selection of datasets can be found in Appendix B.4
Splitting strategy: When not using predefined splits, the train sets are split into two distinct sets in a stratified fashion: a first one to train the different classifiers, corresponding to 40% of the training set and another one to train the trigger model, trained over the 60% left. The set used to train the classifiers is itself split into two different sets in order to train calibrators, using 30% of the given data. Because of this procedure, we have been led to exclude some of the datascts, due to their limited training set size.
All the experiments have been performed using a Linux operating system, with an Intel Xeon E5-2650 2.20GHz (24- cores) and 252GB of RAM. Procceding all datasets (including both blue and orange cylinders) over all competing approaches takes between 9-10 days, using MINIRCCKET classifier, which the most efficient tested.
4.2
Experiments with balanced misclassification and linear delay costs
This first setting is the one most widely used in the literature to date.
4.2.1
Cost definition
The misclassification cost is symmetrical and balanced; the delay cost is linear. They can be defined as follows:
Cm(lly) = 1 # v)
t Ca(t) T
6 All original datascts of the paper be downloaded, already prepared and split, from tttppa//uulz... fttlt/ffr//quu
13

---

Published in Transactions on Machine Learning Research (11/2025)
alap
High delay cost
2
asap
calimera
3-
erdire
4
meen rank
5-
cconony
6-
proba_ threshold
7-
stopping rule
8-
Teaser
__ cost.
9
teaser
cost-informed
11
Low delay cost
cost-uminformed
0.1
0.3
0.5
0.7
0.9
alpha
(a) Evolution of the mean ranks, for every a, based on the AugCost metric. Shaded areas correspond 90% confidence intervals.
6
5
3
2
ects
0.42
0.23
stopping rule
eodire 0.36
0.23
proba_ threshold
teaser hm 0.26
0.23
economy
DODC 0.29
0.24
calimera
teaser
avg cost 0.25
(b) Alpha is fixed
0.5. Wilcoxon signed-rank test labeled with mean AugCost
Figure 3: The ranking plot (a) shows that, across all values of top group of four approaches distinguishes itself. The significance of this result is supported by statistical tests. Specifically, we report this for a 0.5 as shown in (b).
Thus, for each dataset, the AugCost, bounded between and 1, as, within this cost definition, the average misclassification (resp. temporal) cost, across examples equivalent the 1 Accuracy (resp. Earliness) measure, with Accuracy = MM 1(9 = y:) and Earliness = MXX DM &.
4.2.2 Results and analysis
For comparability reasons, this first set of experiments is analyzed over the classical ECTS benchmark used in the literature so far (blue cylinder in Figure Results over the new, non z-normalized, datasets can be found in Appendix
Figure 3a allows for broad look, varying the relative costs of misclassífication and delaying prediction using Equation where small value of means that delay cost is paramount. 90% level confidence intervals have been computed using bootstrar Again, the same four methods top the others for almost every value of a. Not surprisingly, the baseline ASAP (predict as soon as possible) is very good when the delay cost is very high, while ALAP (predict at time T) is very good when there is no cost. associated with delaying decision.
When evaluated by their average rank on all data sets with respect to the average cost (Equation 9). here for = 0.5, four methods significantly outperform the others:
RResampee with replacement. has been done large number of times (10.000x) and are reported as shaded colors in the figure. The statistic of interest is studied, here the mean, examining the bootstrap distribution the desired confidence level.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Leading ECTS methods and their propertics along dimensions underlined in the taxonomy.
Methods
Confidence
Anticipation
Cost-informed
STOPPING RULE
PROBA TURESHOLD
ECONOMY-y-MAX
(test)
CALIMERA
Figure provides a view about the relative performances of the tested methods in terms of the average cost induced using the methods for 0.5. The Wilcoxon-Hoom Ranked test provides an overall statistical analysis. It examines the critical difference among all techniques to plot the method's average rank in horizontal bar. Lower ranks denote better performance, and the methods connected by horizontal bar are similar in terms of statistical significance.
It is remarkable that, in this cost setting, the simple PROBA THRESHOLD method exhibits a strong performance for almost all values of a. It is therefore worth including in the evaluation of new methods. However, while Figures 3a 3b are useful for general analysis, they do not provide insights about how the Accuracy vs. Earliness trade-off is optimized for each of the competitors. Figure g provides some explanation for this.
0.80
0.80
0.78
0.75
0.76
pareto frontier
0.70
0.74
0.8
accuracy
calimera
0.3
0.4
0.5
cedire
0.65
0.58
economy
0.60
proba threshold stopping_ rule leaser avg cost teaser hm
0.56
0.55
cost-informed cost uninformed
0.54 a =0.2
0.1
0.2
0.3
0.4 earliness
0.5
0.6
0.7
0.8
0.1
Figure 4: Pareto front, displaying for each a the Accuracy on the y-axis and Earliness on the r-axis. Best approaches are located on the top left corner. In zoomed boxes, on the right of the Figure, points corresponding to a single a are highlighted, while other points are smaller and gray. Each of the trigger model is optimizing the trade-off in its own way, resulting many different approaches having points in the Pareto dominant set.
In this figure, the two evaluation measures: Accuracy and Earliness, considered dimensions in conflict. The Parelo front is the set of points for which no other point dominates with respect to both Accuracy and Earliness. It is drawn here when varying their relative importance using (in the set {0, 00,, 0.2,...,1.00),,
One must note first that, as ECTS and ECDIRE are cost-uninformed, their performance docs not vary with a. Whatever the relative weight between accuracy and earliness, they make their prediction approximately after having observed half of the time series and they reach an average accuracy respectively near 0.64 and 0.77. They are clearly dominated by the other methods. This is also the case for TEASER HM, which, while being cost-informed, only appears once in the figure. Indeed, weighting mechanism is provided in the original version of the algorithm, where the harmonic mean is used optimization criterion (Shääeer & Leser 2020
15

---

Published in Transactions on Machine Learning Research (11/2025)
100
-0
-0.5
trae class O
a
a
80
3.
100r
100p
2-
1
trae
0
0/
penteted rlow
pemsited claee
trode slw
pcoportion soen of the thms scties
(b) Misclassification cost
(a) Exponential delay cost:
matrix
Ca(t) = (1 aa) exp( log(100))
(three class problem)
Figure 5: Representative delay cost (a) and misclassification ones (b) for an anomaly detection scenario. In our experiments, E 10.
Each of the leading methods STOPPING RULE, PROBA THRESHOLD, ECONOMY and CALIMERA have least one point the Parcto front and generally exhibit combined performance very close to it. A closer look reveals how each approach optimizes the Earliness vs. Accuracy trade-off differently for fixed cost. If we consider example, it appears that ECONOMY takes its decision earlier than PROBA THRESHOLDD itself being more precocious than ECEC. Because this is also an area of problems where the delay cost is low, by doing so, ECONOMY prevents itself from benefiting from waiting for more measurements and increasing its performance. Hence its slight downward slope on Figure 3a for high values of a.
0.8, for
It worth noting that the two naive baselines ASAP and ALAP perform better than the majority of approaches on seven values out of ten. This is especially the case when the delay cost is large, i.e. for E [0.1, 0.3), for which the ASAP baseline is as competitive as top performers. Globally, the performance of PROBa Threshold is remarkable in this cost sctting. Even though it is simply based on single threshold on the confidence in the current prediction, its performance makes it one of the best methods.
The results computed over the proposed datasets ensemble (i.e. orange cylinder) are displayed in Figure 10 Appendix F No significant. changes can be observed in the ranking of competing approaches.
4.3 Experiments with unbalanced misclassification and non-linear delay costs
While the previous section has provided a first asscssment how the various methods adapt to diffcrent respective weights for the misclassification and the delay costs, it nonetheless assumed that the misclassification costs were balanced (e.g. 0 if correctly classified and otherwise) and that the delay cost was linear function of time.
There are, however, applications where these assumptions do not hold, for instance, predictive maintenance hospital emergency services, are characterized by (1) unbalanced misclassification costs (e.g. it is more costly to have to repair machine than to carry out a maintenance operation that turns out not being necessary) and by (ii) non-linear delay costs (e.g. usually, the later the surgical operation is decided, the costlier it is to organize it and the larger the risk for the patient). the following, we call all applications presenting these characteristics "anomaly detection" applications.
The question arises to how the various ECTS algorithms behave in this case, depending on their level of cost awareness and whether or not they are anticipation-basse based. This is what is investigated in the series of experiments reported in this section.
16

---

Published in Transactions on Machine Learning Research (11/2025)
alap
High delay cost
2-
asap
calimera
3-
endire
4
mean rank
5-
cconony
6-
proba_ threshold
7-
stopping rule
8-
Teaser __ cost.
9
teaser
cost-informed
11
Low delay cost
cost-uminformed
0.1
0.3
0.5
0.7
0.9
alpha
(a) Evolution of the mean ranks, for every a, based on the AugCost metric. Shaded areas correspond 90% confidence intervals.
6
5
3
2
ects 16.28 28
6.21
cconomy
eodire 20.24
6,20
calimera
teaser hm 7.70
6.35
stopping rule
DODC
14.71
6.91
proba threshold
teaser
avg cost 7.35
(b) Alpha is fixed
0.5. Wilcoxon signed-rank test labeled with mean AugCost
Figure 6: The ranking plot (a) shows that, across all a, top group composed by three approaches distinguish. This result is significant as supported by statistical tests. Specifically, for a 0.5 as shown in (b).
4.3.1 Cost definition for anomaly detection
In order study the behavior of the various algorithms on scenarios corresponding to anomaly detection, we set the unbalanced misclassification cost matrix such that false negative (i.e. missing anomaly) was 100 times costlier than false positive (i.e. wrongly predicting anomaly) (see Figure 5b). For this last situation, the delay cost was arbitrarily set to 1. The delay cost is defined as an exponential function of time. In order to have a delay cost commensurable with the misclassification one, we decided that waiting for the entire time series to be seen, al T, would cost 100 x a (see Figure 5aj, starting at (1 a) for t = 0 and reaching 100 x a when t= T.
4.3.2 Results and analysis
In this part, Thus, we choose to use the new non z-normalized datasets collection (orange cylinder in Figure 2 In order for the imbalanced misclassification cost to make sense, the datasets have been altered so that the minority class represents 20% of all labels. As explained in Section 4.1 some extrinsic regression datasets are turned into classification ones. In these cases, the threshold value has been set to the second decile the regression target. For the original classification datasets, the minority class has been sub-sampled when necessary.
cost setting is explored, there need to produce comparable results from previous works.
Results from the Wilcoxon-Hollm Ranked test (both regarding the average rank and the value for AugCost) (see Figure 6b) and from the AugCost plot (see Figure (a) with varying values of (in Equation 9 show that now, the best method overall is ECONOMY which is cost-informed at testing time, in addition to being anticipation based. However,
17

---

Published in Transactions on Machine Learning Research (11/2025)
STOPPING-RUL is very strong contender while being cost-informed but not at testing time and confidence-basee There is a reason for it. When STOPPING RULE equals or overpasses ECONOMY, this applies to high values of a when the delay cost loses its importance, therefore leaving the misclassification cost to reign and confidenco-based methods be effective.
It may come a surprise that CALIMERA lags behind ECONOMY for a E [0, 0.4], despite being similarly based on the estimation of future cost expectations. One reason for this is that the cost expectation is achieved by considering only the predicted class. This poor estimate of the cost expeclancy becomes critical when the delay cost is important.
cost
Similarly, PROBA THRESHOLD is surprisingly good in this scenario, even if it is no longer in the top tier. Looking solely at prediction confidence, we might expect it to be blind to the rapid increase in delay cost the anomaly detection scenario. However,
noticeable that the cost of delay only increases sharply afler around 60% of the complete time series has been observed, which is generally sufficient exceed the confidence threshold. Hence, PROBA THRESHOLD does not sulfer from high delay costs that are come, and exhibits good performance here.
Figure f plots the Pareto front, considering two axes based on decision costs. The horizontal axis corresponds to the average delay cost incurred for each example, normalized by the worst delay cost paid at = T. It is better to be on the left of the r-axis. The vertical axis corresponds to one minus the misclassification cost incurred for each example, normalized by the worst prediction cost. It is better to be high on the y-axis.
We observe that the Pareto front is composed almost exclusively of points corresponding the ECONOMY and CALIMERA methods. This is consistent with the evaluation based on the AvgCost metric. This figure highlights the fact that the design of approaches capable of handling arbitrarily parameterized decision costs requires cost- informed application framework.
0.6
0.5
normolhed meen misll cost
Figure Pareto front, displaying for cach a, the normalized version of the AvgCost, decomposed over delay and misclassification cost r-axis and y-axis respectively. Best approaches are located on the top left cor- ner. Due to the exponential shape of the delay cost, the r-axis is on log scale.
0.4
pareto frontier
calimera
cedire
ecec
0.3
ccunomy
ools
proba threshold
stopping rule
0.2
cost-informed
teaser avg cost
cost-uninformed
traser hm
033
0.5
normalized mean delay
4.4 Other experiments: ablation and substitution studies
In this section, complementary experiments, namely ablation studies as well as sanity checks are briefly discussed. For the sake of brevity, the figures supporting the analysis are reported in Appendix
Impact of removing calibration
Bilski & Jastrzebska (2023) assert that calibration of the classifiers is paramount for the performance of ECTS algorithms. In order to test this claim, have repeated the experiment, removing the calibration step. The examples used for calibration have also been removed during training, so that all else remains the same as before.
18

---

Published in Transactions on Machine Learning Research (11/2025)
The results of Figure 13 in Appendix F.3 show that indeed CALIMERA suffers greatly if no calibration is done. Indeed, this approach relies on estimating the expectation of future costs via a regression problem, and miscalibration may have a nogative impact on the built targets. For its part, PROBA THRESHOLD suffers somewhat mildly. This is no surprise, as they rely on a single threshold on the confidence of the prediction for all time steps.
Impact of removing non-myopia
Experiments have shown that anticipation--aaee approaches tend to outperform the myopic competitors; we further validate this by conducting an ablation study over the non-myopia of both ECONOMY and CALIMERA.
property is critical, whereas myopicc counterparts significantly fall behind original methods. Intuitively, for higher temporal costs, the anticipatiion-based property is less important decisions tend to be taken earlier, and it is less necessary for the methods to anticipate the future.
Figure 14 in Appendix F F.4 clearly shows that, for weak temporal costs, the anticipation-
Impact of the choice of base classifier
All methods have boen compared using the same classifier: MINIROCKET so that only the decision components differ. However, the choice of the base classifier could induce a bias favoring hampering some methods. In order
to clarify this, we have repeated the experiments replacing MINIROCKET with two base classifirs: WEASEL 2.0
(Schäfer & Leser 2023), and the XGBoosT classifier (Chen et al. 2015) using features produced by TSFRESH Christ et al.] 2018). Both of these classifiers have already been lested within the ECTS literature by [Schffer Leser 2020
L.v et al. (2019) and AAchenchbee et al. (2021) respectively. Figure 15 and 16 in Appendix F.5 report the results respectively with these two classification methods. One can observe that the results are not significantly altered with the same overall ordering of the methods when varying the value of a. Furthermore,
results on AugCost show
that performance tends to be belter for all methods using MINIROCKET (see Table in Appendix F.5) It is thus to be preferred given its simplicity and good performance.
Impact of z-normalization
Considering the newly proposed ensemble of datasets, were not able to identify any problems of information leakage over time. This inconclusive result simply indicates that the variance of the time series measurements is not informative for these datasets, which still could be the case considering past published results. For further details, please refer to Appendix F.6
5 Conclusion
In this paper, we have proposed taxonomy that allows to underline the main familics of separable approaches for the ECTS problem, pointing out the essential components and the questions that have to be tackled when designing new method. We have thus enlightened (i) the importance of the two components: decision and prediction, (.) the distinction between anticipation-buase based and myopic methods, and (Mii between cost-informed and cost-uninformed techniques.
We have defined a methodology for evaluating and comparing ECTS methods. In addition, and we hope this will prove useful to the community, we have built an open source library that includes systematic implementation of the methods tested, the proposed evaluation protocol, as well as a collection of 34 datasets designed to enable informative testing of the methods.
We have also underlined the importance of considering variety of cost settings in the evaluation of ECTS methods so as to reflect what can happen in real-life applications.
The in-depth experiments carried out shed light on design choices. Firstly, are anticipaaion-bused methods better than myopic ones? We have shown that four of the eleven methods tested perform significantly better. However, these methods do not share the same characteristics with regard to this question. Secondly, to the question: do cost-informed methods outperform cost-uninformed methods? The Pareto front indicates a clear superiority of the former. Finally, when test the robustness of methods to variations in cost functions, anticipation- based methods prove superior and, among them, cost- methods fare even better.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Our experiments have also shown that calibration of the classifiers has large impact on some methods (e.g. CAL- IMERA) in particular, less so on other methods (e.g. PROBA THRESHOLD and ECDIRE). As regards to the z- normalization of time series which would be unwisely used in evaluation studics, our results (see Appendix F. show that the impact on the performance is limited, thus suggesting that the existing trigger functions do not or only slightly benefit from this information leakage.
Future work could be carried out to study the literaturo's 's approaches applicd in yet unezplored cost settings. For example, in many applications, the delay cost depends on the true class and the predicted one, and thus a single cost function integrating misclassification and delay costs should then be used. This general cost form requires the adaptation of some statc-of-tho-art methods and has not yet been studicd.
In addition, in real ECTS applications, it is up to the business expert to define the costs, which not an easy task in practice. Among the challenges, applications where the costs actually paid are not deterministic are of key interest (e.g. manufacturing defect on an engine part does not necessarily lead to a failure, but it does increase the probability of paying a higher cost). Thus, future work could study the impact of stochastic cost functions. Another interesting case is applications where the costs paid are slightly different, or changed, from those defined by the business experts for the training phase (e.g. a change in the price of raw materials). Those kinds of cost drift between training and testing stages could also be further studied.
Finally, the case of existing separable approaches, the misclassification cost is not exploited for training the classification function. Future work could investigate the interest of using cost-sensitive classifiers in the case of ECTS.
Beyond separable approaches, more extensive review of end-to-end ECTS methods could be interesting to complete the one presented in this paper. In particular, an experimental protocol should be designed to place end-to-end and separable methods on similar ground and to design new, efficient approaches.
References
Youssef Achenchabe, Alexis Bondu, Antoine Cornuéjols, and Asma Dachraoui. Early classification of time series: Cost-based optimization criterion and algorithms. Machine Learning, 11006))148111104, 2021.
Charilaos Akasiadis, Evgenios Kladis, Petro-Foti Kamberi, Evangelos Michelioudakis, Elias Alevizos, and Alexander Artíkis. A framework to evaluate early time-series classification algorithms. In EDBT, pp. 623-633- 2024.
Carlos J Alonso González and Juan J Rodríguez Diez. Boosting interval-based literals: Variable length and early classification. In Data mining in time series databases, pp. 149-171. World Scientific, 2004.
Hyrum S Anderson, Nathan Parrish, and Maya R Gupta. Farly time series classification with reliability guarantee. Technical report, Sandia National Lab. (SNL-NM,,, Albuquerque, NM (United States), 2012.
Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021.
Alessandro Antonucci, Mauro Scanagatta, Denis Deratani Mauá, and Cassio Polpo de Campos. Early classification of time series by hidden markov models with set-valued parameters. In Proceedings of the NIPS time series workshop, pp. 1-5, 2015.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 31:606-660 2017.
James O Berger. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 1985.
Jakub Michal Bilski and Agnicszka Jastrzebska. Calimera: A new carly time scries classification method. Information Processing & Management, 6055)10034 2023.
Alexis Bondu, Youssef Achenchabe, Albert Bifet, Fabrice Clérot, Antoine Cormuéjols, Joao Gama, Georges Hébrail, Vincent Lemaire, and Frarr-FFançois Marteau. Open challenges for machine learning based early decision-making research. ACM SIGKDD Explorations Newsletter, 24(2):12 33, 2022.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Paul Boniol, Qinghua Liu, Mingyi Huang, Themis Palpanas, and John Paparrizos. Dive into time-series anomaly detection: decade review. arXiv preprint arXiv:2412.20512 2024.
Alexander Cao, Jean Utke, and Diego Klabjan. A policy for early sequence classification. In International Conference on Artificial Neural Networks, pp. 50-61. Springer, 2023.
Huiling Chen, Ye Zhang, Aosheng Tian, Yi Hou; Chao Ma, and Shilin Zhou. Decoupled early time series classification using varied-length feature augmentation and gradient projection technique. Entropy, 24(10):1477, 2022.
Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1 4, 2015.
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (tsfresh-a python packago). Neurocomputing, 307:72-77, 2018
Asma Dachraoui, A Alexi Bondu, and Antoine Cormuéjols. Early classification of time series as a non myopic sequential decision making problem. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part 15, pp. 433 447. Springer, 2015.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Choti- rat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAR Journal of Automatica Sinica, ((6):1293-130)) 2019.
Morris H DeGroot. Optimal statistical decisions. John Wiley & Sons, 2005.
Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. Minirocket: A very fast (almost) deterministic transform for time series classification. In Proceedings of the 27th ACM STGKDD conference on knowledge discovery & data mining, pp. 248 257, 2021.
Akinori F Ebihara, Taiki Miyagawa, Kazuyuki Sakurai, and Hitoshi Imaoka. Toward asymptotic optimality: Se- quential unsupervised regression of density ratio for carly classification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Akinori F Ebihara, Taiki Miyagawa, Kazuyuki Sakurai, and Hitoshi Imaoka. Learning the optimal stopping for early classification within finile horizons via sequential probability ratio test. In The Thirleenth International Conference on Learning Representations, 2025.
Thomas S Ferguson. Who solved the secretary problem? Statistical science, 4(3):282-28) 1989.
Mohamed F Ghalwash and Zoran Obradovic. Early classification of multivariate temporal observations by extraction of interpretable shapelets. BMC bioinformatics, 13:1-12, 2012.
Mohamed F Ghalwash, Vladan Radosavljevic, and Zoran Obradovic. Utilizing temporal patterns for estimating uncertainty in interpretable early decision making. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 402 411, 2014.
Bhaskar Kumar Ghosh and Pranab Kumar Sen. Handbook of sequential analysis. CRC Press, 1991.
Ashish Gupla, Hari Prabhat. Gupta, Bhaskar Biswas, and Tanima Dutta. Approaches and applications of early classification of time series: A review. IEEE Transactions on Artificial Intelligence, 1(1):47 61, 2020.
Thomas Hartvigsen, Cansu Sen, Xiangnan Kong, and Elke Rundensteiner. Adaptive-halting policy network for early classification. In Proccedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 101-110, 2019.
Thomas Hartvigson, Walter Gerych, Jidapa Jardsjarassir, Xiangnan Kong, and Elke Rundensteiner. Stop&chop: Early classification of irregular time series. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 696-705, 2022.
Nima Hatami and Camelia Chira. Classifiers with a reject option for early time-series classification. In 2013 IEEE symposium computational intelligence and ensemble learning (CIEL), pp. 9-16. IEEE, 2013.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Guoliang He, Yong Duan, Tieyun Qian, and Xu Chen. Early prediction on imbalanced multivariate time series. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 1889-1892, 2013.
Guoliang He, Yong Duan, Rong Peng, Xiaoyuan Jing, Ticyun Qian, and Lingling Wang. Early classification on multivariate time series. Neurocomputing, 149:777-787, 2015.
Ali Fawaz, Angus Dempster, Chang Wei Tan, Matthicu Herrmann, Lynn Miller, Daniel Schmidt, Stefano Berretti, Jonathan Weber, Maxime Devanne, Germain Forestier, and Geoff Webb. An approach to multi- ple comparison benchmark evaluations that is stable under manipulation of the comparate set. arXiv preprint arXiv:2305.11921, 2023.
Evgenios Kladis, Charilaos Akasiadis, Evangelos Michelioudakis, Elias Alevizos, and Alexandros Artikis. An empirical evaluation of early time-series classification ion algorithms. In EDBT/ICDT Workshops, 2021.
Achim Klenke. Probability theory: a comprehensive course. Springer Science & Business Media, 2013.
Yu-Feng Lin, Hsuan-Hsu Chen, Vincent S Tseng, and Jian Pei. Reliable early classification on multivariate time series with numerical and categorical attributes. In Advances in Knowledge Discovery and Data Mining: 19th Pacific-Asia Conference, PAKDD 2015, Ho Chi Minh City, Victnam, May 19-22, 2015, Proceedings, Part 19, pp. 199 211. Springer, 2015.
Junwei Lv and Xuegang Hu. A bascline for early classification of time series in an open world. In 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 343 -311. IEEE, 2022.
Junwei Lv, Xuegang Hu, Lei Li, and Peipei Li. An effective confidence-basee early classification of time series. IEEE Access, 7:96113 13 96124,, 2019.
Junwei Lv, Yuqi Chu, Jun Hu, Peipei Li, and Xuegang Hu. Second-order confidence network for early classification of time scries. ACM Transactions on Intelligent Systems and Technology, 2023.
Coralie Martinez, Guillaume Perrin, Emmanuel Ramasso, and Michèle Rombaut. A deep reinforcement learning approach for early classification of time series. In 2018 26th European Signal Processing Conference (EUSIPCO). pp. 2030-2034. IEEE, 2018.
Coralie Martinez, Emmanuel Ramasso, Guillaume Perrin, and Michèle Rombaut. Adaptive early classification of temporal sequences using deep reinforcement learning. Knowledge-Based Systems, 190:105290, 2020.
Chirag Mathukia, WuQQang Fan, Karen Vadyak, Christine Biege, and Mahesh Krishnamurthy. Modified early warning system improves patient safety and clinical outcomes in an academic community hospital. Journal of community hospital internal medicine perspectives, 5(2):26716, 2015.
Matthew Middlehurst, Ali Ismail-Fawaz, Antoine Guillaune, Christopher Holder, David Guijo Rubio, Guzal Bu- latova, Leonidas Tsppronis, Lukasz Mentel, Martin Walter, Patrick Schäfer, al. aeon: a python toolkit for lcarning from time series. arXiv preprint arXiv:2406. 14231, 2024a.
Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. Bake off redux: review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery, pp. 1-74, 2024b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Usue Mori, Alexander Mendiburu, Sanjoy Dasgupta, and Jose A Lozano. Early classification of time series by simultancously optimizing the accuracy and carlinoss. IEEE transactions on neural networks and learning systems, 29( 29(1))44669 4578, 2017a.
Usue Mori, Alexander Mendiburu, Eamonn Keogh, and Jose A Lozano. Reliable early classification of time series based on discriminating the classes over time. Data mining and knowledge discovery, 2017b.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Nathan Parrish, Hyrum S Anderson, Maya R Gupta, and Dun Yu Hsiao. Classifying with confidence from incomplete information. The Journal of Machine Learning Research, 14(1):35011333) 2013.
John Platt et al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3))61-74, 1999.
Oona Rainio, Jarmo Teuho, and Riku Klén. Evaluation metrics and statistical tests for machine learning. Scientific Reports, 14(1):6086, 2024.
Liran Ringel, Regev Cohen, Daniel Freedman, Michael Elad, and Yaniv Romano. Early time classification with accumulated accuracy gap control. arXiv preprint arXiv:2402.00857, 2024.
Marc RuBwurm, Nicolas Courty, Rémi Emonet, Sébastien Lefévre, Devis Tuia, and Romain Tavenard. and Icarned carly classification of time scries for in-scason crop type mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 196:445 456, 2023.
Tiago Santos and Roman Kern. iknow, 2016.
literature survey of early time series classification and deep learning. Sami@
Azusa Sawada, Taiki Miyagawa, Akinori Ebihara, Shoji Yachida, and Toshinori Hosoi. Convolutional neural networks for time-dependent classification of variable-length time series. In 2022 International joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2022.
Patrick Schäfer and UIf Leser. Teaser: early and accurate time series classification. Data mining and knowledge discovery, 34(5):1336-1133) 1362, 2020.
Patrick Schäfer and Ulf Leser. Weasel 2.0: a random dilated dictionary transform for fast, accurate and memory constrained time series classification. Machine Learning, 112((2):4763 4788, 2023.
Shubhranshu Shekhar, Dhivya Eswaran, Bryan Hooi, Jonathan Elmer, Christos Faloutsos, and Leman Akoglu. Benefit-aware carly prodiction of health outcomes on multivariate cog time scries. Journal of biomedical informatics, 139:104296, 2023.
Larry A Shepp. Explicit solutions to some problems of optimal stopping. The Annals of Mathematical Statistics, 40 (3):993, 1969.
Sergii Skakun, Belen Franch, Eric Vermote, Jean-Claude Roger, Inbal Becker-Reshef, Christopher Justice, and Na- taliia Kussul. Early season large-area winter crop mapping using modis ndvi data, growing degree days information and gaussian mixture model. Remote Sensing of Environment, 195:244 258, 2017.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I Webb. Monash university, uea, ucr time series extrinsic regression archive. arXiv preprint arXiv:2006.1099 2020.
Alexander Tartakovsky, Tgor Nikiforov, and Michele Basseville. Sequential analysis: Hypothesis testing and changepoint detection. CRC press, 2014.
Romain Tavenard and Simon Malinowski. Cost-aware early classification of time series. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16, pp. 632-647. Springer, 2016.
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Noural networks, 22(6-)):544 557, 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Abraham Wald and Jacob Wolfowitz. Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, pp. 229, 1948.
Wenlin Wang, Changyou Chen, Wenqi Wang, Piyush Rai, and Lawrence Carin. Earliness-aware deep convolutional networks for early time series classification. arXiv preprint arXiv:1611.04578, 2016.
23

---

Published in Transactions on Machine Learning Research (11/2025)
Yifan Wang, Qining Zhang, Lei Ying, and Chuan Zhou. Deep reinforcement learning for early diagnosis of lung
cancer. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 38, pp.
2024.
Renjie Wu, Audrey Der, and Famonn Keogh. When is early classification of time series meaningful. IEEE
Transactions Knowledge and Data Engineering, 2021.
Zhengzheng Xing, Jian Pei, Guozhu Dong, and Philip Yu. Mining sequence classifiers for early prediction. In Proceedings of the 2008 SIAM international conference on data mining, pp. 644 655. SIAM, 2008
Zhengzheng Xing, Jian Pei, and S Yu Philip. Early prediction on time series: A nearest neighbor approach. In IJCAL, pp. 1297-1302. Citescer, 2009.
Zhengzheng Xing, Jian Pei, Philip S Yu, and Ke Wang. Extracting interpretable features for early classification on lime series. In Proceedings of the 2011 SIAM international conference dała mining, pp. 247 258. SIAM, 2011.
Zhengzheng Xing, Jian Pei, and Philip Yu. Early classification on time series. Knowledge and information systems. 31:105-10 127, 2012.
Wenhe Yan, Guiling Li, Zongda Wu, Senzhang Wang, and Philip S Yu. Extracting diverse-shapelets for early classification on time series. World Wide Web, 23:3055-30835 2020.
Liuyi Yao, Yaliang Li, Yesheng Li, Hengtong Zhang, Mengdi Huai, Jing Gao, and Aidong Zhang. Diec: Distance transformation based early time series classification. In Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 486-494. SIAM, 2019.
Lexiang Ye and Eamonn Koogh. Time series shapelets: novel technique that allows accurate, interpretable and fast classification. Data mining and knowledge discovery, 22:149 182, 2011.
Paul-Emile Zafar, Youssef Achenchabe, Alexis Bondu, Antoine Cornuéjols, and Vincent Lemaire. Early classification of time series: Cost-based multiclass algorithms. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), pp. 1-10. TEFE, 2021.
Wenjing Zhang and Yuan Wan. Early classification of time series based on trend segmentation and optimization cost function. Applied Intelligence, pp. 1-12, 2022.
24

---

Published in Transactions on Machine Learning Research (11/2025)
A Benchmarking: related studies
Table 3: Table of published methods for the ECTS problem with their benchmarking protocol.
References
No. of datasets
No. of methods
Stats test
Eval. metric(s)
Cost balance Code
Reject Hstami Chira
2013
Aca,/Barl
fixed fixod
iHMM Antomooci et al
2015
2
fon1 Acc,
ECDIRE Moni
2017b
fixed
Stopping Rule Mori et al 2017a
45
5
Acc, Acc./Earl.
vary
ECEC Lv 2019
45+8
Acc. /Farl.
vary
TEASER Schäfer & Leser
2020
45
Ace, Ace./Earl./ /Erl//HM
vary
SOON Lv 2023
45
Ace,/ Acee//Erll//Ih
fixed
ECTS
Xing et 2012
23
Aeal/Earl.
fixod
ReIClass
Parrish et al.
2013
15
2
Acc./Farl.
vary
2step/ NhCtster
Tavenard & Malinowski
2016
AugCost/Aoo
vary
ACONOMY-a-mann Zafar et al.
2021
15/31
2
AugCost
CALIMERA Bilski & Jastrzebske
2023
45/34
AvgCost/ Acc. hoce//Eel,//tn
fixed
FIRMBOUND Ebilara et al.
2025
AAPR/SAT
EDSC Xing
2011
Aco./Eatl
fixod
EARLIEST Hartvigsen al.
2019
3
Ace,/ (arl
DDQN Martinez
2020
Acc,/Earl.
vary
DETSCNet Chen et al.
2022
12
5
HM
fixed
Benefitter
Shekhar
2023
Acr./Farl.
vary
CIS Cao et sl.
2023
3
Ace,/
fixed
ELECTS Rulwurm
2023
Aco./Eart
vary
EarlyStop-RL. RL WWing et al. 2024
FPR/FNR /
fixed
Empirical Survey Akasiadis et al. 2024
Acc,/Eat /HM///
fixed
Columns defined as
No. of datasets: the number of datascts used in the paper.
No. of methods: the number of methods tested in the paper (including the one proposed, any).
Stals lest: whether evaluation has been performed using statistical tests [Ranni et al. 2024
Eval. metric(s): evaluation metrics used.
Cost balance: whether experiments have been conducted for different cost balances between Cm and Ca.
Code: is the code the method available.
Experimental methodology is heterogeneous in the literature so far, making comparisons between methods more difficult. When considering datasets used or evaluation metrics, a variety of different approaches can be found, as shown in Table 3
B Experimental protocol
B.1 Optimization of the parameters of the methods
Eight methods from the literature have been tested, respecting far as possible the choices made in the original papers. Two groups of hyperparameters need to be set: (t) some of them meta parameters independent of the
25

---

Published in Transactions on Machine Learning Research (11/2025)
dataset and have been fixed according to the original papers, (ii) others have to be optimized using grid search based on the AugCost criterion. The optimization of the second group of hyperparameters has been carried out using the value bounds mentioned in the originally published papers. When possible, the granularity of the grid has been adapted to keep similar computation times between competitors. These two groups of hyperparaneters are described for all methods in the Appendix [.2 As a remark, because the original version of TEASER uses the harmonic mean (Schäfer & Leser 2020), have kept this setting (the resulting method being TEASER and we have added a variant called TEASER optimized using AugCost.
B.2 Hyperparameters
Table 4: Hyperparameters' value. A defines grid's 'ss size when performing grid-search over continuous valued intervals.
Hyperparameters
Method
Fixed
Optimized
CALIMERA
kernel: 'rbf"
ECDIRE
pere acc 100% support
ECTS
ECONOMÝ
1: [1 20]
PROBA THRESHOLD
A 40
71, 72, 73 E [-1,]]
STOPPING RULE
A 10
TEASER
[1 ..5]
B.3 Calibration of the classifications
Like Bilski Jastrzebska (023),, we add calibration step when learning the classifiers, i.e. Platt's scaling (Platt ct al. 1999). Indeed, we dealing with collections of independently trained classifiers, the prediction scores may not remain consistent with one another over the time dimension. However, the trigger methods usually have their parameters set with the same values for all time steps. This is the case, for example with the PROBA THRESHOLD approach. In addition, some approaches such as CALIMERA and ECONOMY-7-MAX exploit the estimated posterior probabilities {p(y|x))ye) to estimate the future cost expectation. It is therefore highly desirable for all classifiers, at all times, to have their output calibrated and is necessary for a fair comparison.
B.4 Datasets selection
When a time series does not bring increasing level of information over time about its class, classifiers likely to obtain the same confusion matrix for all time steps, and the trigger function should then choose to make a prediction at the first time step to avoid the delay cost even if the prediction is very uncertain. Although this case may occur, it cannot form the basis for comparing the trigger functions. This is why we have been careful about the selection of data sets.
We have chosen data sets where the classification performance tends to increase over time, as measured with the same basis classifier for all methods (see Section (..tio We check that the train AUC increases averaged over a time window of size 5, when using either the beginning of the series, almost complete ones. Specifically, the beginning of the series is laken using 5%... 25% and the almost complete series are of the following lengths 75%, ..., 100%.
When the difference of averaged AUC is strictly positive, the dataset is selected. (see Table 7 Tn addition, all datasets exhibiting z-normalization have been removed. Thirty-four datasets passed these demanding criteria (see
Appendix E
For instance, Figure g (left) displays the time series of synthetic data set where each of the three classes is characterized by specific pattern over time. Class 0 can thus be recognized early on, while class and class 2 only be discriminated after approximately one third of the length of the time series. This is confirmed by the accuracy of the recognition of each class (right).
26

---

Published in Transactions on Machine Learning Research (11/2025)
Selecting data sets appropriate for the comparison of ECTS methods is demanding process that must check for all the desirable properties described above. We thus hope that the corpus thus built and made available on the open source library will be useful to the scientific community for future performance studies.
1.0
0.5
Class
0.0
Class
accuii
0.6
0.5
0.0
0.4
Accuracy, class
1.0
Accuracy, class
Accuracy, class
0.5
Class 2
Global Accuracy
10
10
timestamps
timestamps
Figure Examples of training time series the SmoothSubspuce dataset. Each the class has a discriminative signal within one third of the serie (see Appendix D for a code snippet reproducing these results, based on the proposed library).
C
End-to-end ECTS
C.1
State-of-the-aa
Table 5: Table of published end-to-end methods for the ECTS problem.
References
Architecture
RL
DL
Cost-genericity
EDSC Xing et al. 2011
Shapelet
EARLIEST Hartvigsen et al.
2019
LSTM
/
DDQN Martinez et al.
2020
MLP
V
DETSCNet Chen et al.
2022
TCN
Benefitter Shekhar et al.
2023
LSTM
CIS Cao al. 2023)
LSTM
ELECTS Ruswurm et
2023)
LSTM
EarlyStop-RL (Wang et
2024
MLP
A different class of methods relies on searching telltale representations of subsequences, such that if the incoming time sequence matches one or more of these representations, then its class can be predicted. Typically, these representations take the form of shapelets that discriminate well one class from the others (Ye & Kcogh 2011). For instance, the Early Distinctive Shapelet Classification (EDsC) method learns a distance threshold for each shapelet, based on the computation of the Fuclidean distance between the considered subsequence and all other valid subsequences in the training set (Xing et al. (001)). It selects subset of them, based on a utility measure that combines precision and recall, weighted by the carliness. A prediction is made shapelets well enough. Because this family of methods is computationally expensive, extensions have been developed
X; matches one of these
27

---

Published in Transactions on Machine Learning Research (11/2025)
to reduce the computational load Yan et al. 2020;
Zhang & Wan 2022). Other extensions aim at improving the
reliability of the predictions (Ghalwas wash et al. 2014 Yao et al. 2019) and tackling multivariate time series (Ghalwash)
& Obradovic 2012 Hc et al. 2013 2015
Lin ct al 2015
The EARLIEST (Farly and Adaptive Recurrent Label ESTimator) RNN architecture (Hartvigsen et al. 2019 to make the prediction and Reinforcement Learning agent trained jointly using policy gradient to trigger prediction or not. If prediction is triggered, the hidden representation given by the RNN is sent to a Discriminator, whose role is to predict a class, given this representation. The model has been adapted to deal with irregularly sampled time series (Hartvigsen et al. 2022
2020 use Deep Q-Network (Mnih al. 2015 alongside specifically designed reward signal, encouraging the agent to find good trade-off between earliness and accuracy. Those types of approaches also naturally extend to online settings where time series not of fixed length.
Martinez et al. 2018
The Decouple ETSC Network (DETSCNET) (Chen et al. 2022 architecture leverages a gradient projection technique in order to jointly learn two sub-modules: one for variable-length series classification, and the other for the early exiting task.
The BENEFITTER algorithm (Shekhar et al. 2023 is an anticipation-based approach that learns to predict the benefit of triggering prediction early. This quantity is equal to the saving one could make by triggering some decision now minus the cost induced by wrong prediction. A LSTM model is learned to regress the benefit, which thus triggers, at inference time, as soon as the benefit is positive, i.e when savings induced by temporal costs exceed estimated misclassification costs.
The Classifter-nduueed Stopping (CIS) Cao et al. 2023) model leverages policy gradient Reinforcement Learning in order to directly predict the optimal stopping time in a supervised way, found posteriori for the training time series.
The End-to-end Learned Early Classification of Time Series method (ElECTS) leverages an LSTM architecture, adding stopping prediction head the network and adapting the loss function promote good early predictions RuBwwurm al. 2023
Wang et al. (2024) introduce EARIYSTOP-RI,, in which model-free RL is used to address the problem of early diagnosis of lung cancer.
C.2 End-to-end experiments
In this section, we test out two popular ECTS end-to-end methods, iee. FARLIEST (Hartvigsen et al. 2019) and ELECTS (Rubwurm al.] 2023), and directly compare them to the separable baseline PROBA THRESHOLD. Due the fact that those method do not directly expand to generic cost settings, only perform this experiment using
the standard cost setting, described in Section 4.2
28

---

Published in Transactions on Machine Learning Research (11/2025)
High delay cost
EARLIEST ELECTS
mean rank
proba threshold
Low delay cost
3
0.6
0.7
alphe
Figure 9: Standard cost setting, non z-normalized proposed datasets (orange cylinder). Tt is found that end-to- end approaches outperform the separable baseline PRODA THRESHOLD for high temporal costs. Since end-to-end approaches are not constrained to consider time series only over the 20 timestamps corresponding to every 5% of the time series duration for prediction, they make their prediction in the interval. For instance, before 5% of the time series has been observed, which happens for high delay cost, whereas separable methods, in this experimental setting, must wait until the first time step, resulting in higher average costs. However, the delay cost decreases, end-lo-end methods tend to be less stable overall, falling behind simple separable base method, such as PROBa THRESHOLD.
D Library
More details are given about the proposed library here. leverages the modularity of separable ECTS approaches and offers the possibility to easily assess each of the component contribution to final performance. The main library object is EarlyClassifier object that needs, at least, 3 inputs:
a Chronologicallllassifiie that outputs classification prediction based on varying-size time series,
TriggerKodel that decides to whether accept not the current classificr's prediction,
CostMatrices that defines the cost functions to be optimized.
As used throughout the paper, one way implement classification strategy is to use an ensemble of classifiers, each of which specialized for particular timestamp. The ClassifiersColletionn object offers interface to
do that. It takes input any scikit-learn estimator well as set of timestamps for which to learn a classifier. For example, the following code uses the same classification module used to produce Figure g
from sklearn. linear model import RidgeClassifiercy from sklearn, preprocessing import SplineTrranfformrr from sklearn. calibration aalearn.cllibration inport CalibratedClamssfierrr from sklearn, pipeline import make_pipeline
eeleain.
edm. claailfication.cleeiffeee collecttiono import ClasiffirrSolllettioo
series' length
valid timestamps (iangea,,,,,, T+1)
elf make pipelinel Sp1ineTransformer())
29

---

Published in Transactions on Machine Learning Research (11/2025)
Callertccchasssrrrrvviiieeeeeeee "
collection_cle clf Classsflers@ollectiioal base cl
tinestamps-valid, timestamps)
Then, having defined classification strategy, one can fit a full ECTS model, for example, using the ProbabilityThresooldd trigger model. By default, when no other argument than timestamp is given to the CostMatrices object, it uses the default cost setting, as defined in Section 4.2
from ml al__ddneedl caanheerll___eeeee classifier import BarlyClansifier from ml_edm. trigger import Probability/hreehoold from ml aeaddncost_matreeee import CostMatrices
early_clz clf EarlyClassifter(
clf,
trigger htt.r//mmoooooallla/ooooaoaoaddd timesaanss) cost
early_clf wlw.y_clf.ft. firt
Thus, the library allows to easily pursue research in the separable ECTS field, facilitating the implementation of new approaches (whether in the classification trigger part) and the systematic comparison with previous ones. It also pave the way for end-to-end approaches that can be framed as a trigger model that does not rely on any classifier's outputs.
E
Data description
E.1
UCR Time Series Classification datasets
Table UCR TSC datasets 77 datasets from the UCR archive have been relained to run the experiments over the 128 contained the full archive. Those are the ones with fixed longth, without missing values and with enough training samples execute our experiments pipeline end-to-end. Itatic datasets are not included in experiments using default split for this reason.
Train
Test
Length
Class
Тype
Data
ACSF1
100
100
1460
10
Device
Adiac
390
391
176
37
Image
Beef
30
30
470
Spectro
BeetleFly
20
20
512
Image
BME
30
150
128
3
Simulated
Car
60
60
577
4
Sensor
CBF
30
900
128
Simulated
Chinatown
20
345
Traffic
ChlorineConcentrationn
467
3840
166
Sensor
ToCCECGTorso
40
1380
1639
Sensor
Coffee
28
28
286
2
Spectro
Computers
250
250
720
2
Device
X
390
390
300
12
Motion
Cricket Y
390
390
300
12
Motion
CricketZ Crop
390
390
300
12
Motion
7200
16800
24
Image
30

---

Published in Transactions on Machine Learning Research (11/2025)
DiatomSizelldduction
16
306
345
Image
DistalPhallaxxOutinnCCoooo
600
276
Image
Earthquakes
322
139
512
Sensor
ECG200
100
100
96
ECG
ECG5000
500
4500
140
ECG
ECGFiveDays
23
861
136
2
ECG
ElevtricDevices
8926
7711
96
7
Device
EOGVorticaSSignal
362
362
1250
12
EOG
EehenolLevel
504
500
1751
Spectro
FaceAll
560
1690
131
14
Image
FaceFFour
24
88
350
Image
FacesUCR
200
2050
131
14
Image
Fifty Words
450
455
270
50
Image
Fish
175
175
463
7
Image
FordA
3601
1320
500
Sensor
FreezerRegular Train
150
2850
301
Sensor
GunPoint
50
150
150
Motion
Ham
109
105
431
Spectro
HandOutlines
1000
370
2709
Image
Haptics
155
308
1092
5
Motion
Herring
64
64
512
2
Image
HouseTwwntty
34
101
3000
Device
InlineSkate
100
550
1882
Motion
InscctEPGoguular Train
62
249
601
EPG
Insect WingbeeWggg
220
1980
256
11
Sensor
ItalyPowerDemandd
67
1029
24
Sensor
LangeKiitchen&Appllinees
375
375
720
3
Device
Lightning2
60
61
637
2
Sensor
Lightning7
70
73
319
7
Sensor
Mallat
55
2345
1024
Simulated
Meat
60
60
448
Spectro
Medicallmages
381
760
10
Image
MelbournePadestrran
1200
2450
24
10
Traffic
500
2425
1024
5
Image
MoteStrain
20
1252
84
2
Sensor
NoulnvesivePecallCCCCTToaaa
1800
1965
750
42
ECG
1800
1965
750
42
ECG
OSULeaf
200
242
427
Image
OliveOil
30
30
570
Spectro
PhalangesOulliieSCCorr
1800
858
80
Image
Plane
105
105
144
7
Sensor
PowerCons
180
180
144
2
Power
ProximalP ProximalPhalanxOutte balannOOttinneooree
600
291
80
2
Image
Refrigerationeviioss
375
375
720
Device
Rock
20
50
2844
Spectrum
Screen Sype
375
375
720
Device
SemgHHandGenndeChh2
300
600
1500
Spectrum
Shapes.All
600
600
512
60
Image
SmoothSubspace
150
150
15
3
Simulated
Son AAIBORobttSuraceel
20
601
70
2
Sensor
Sony AonAAABBORoooSSsacccc
27
953
65
Sensor
StarLighttCurves
1000
8236
1024
Sensor
Strawberry
613
370
235
Spectro
31

---

Published in Transactions on Machine Learning Research (11/2025)
SwedishLeaf
500
625
128
15
Image
Symbols
25
995
398
Image
SyntheticControl
300
300
60
Simulated
ToeSegmentation1
40
228
277
Motion
Trace
100
100
275
4
Sensor
TwoLedEEGG
23
1139
82
2
ECG
1000
4000
128
4
Simulated
UMD
36
144
150
Simulated
UWaveGesttrelalbaayyy
896
3582
315
Motion
UWaveGessureLLibrryy
896
3582
315
Motion
UWaveGestuueLiirraryy
896
3582
315
Motion
Wafer
1000
6164
152
Sensor
Wine
57
54
234
2
Spectro
WordSymonyms
267
638
270
25
Image
Worms
181
77
900
Motion
Yoga
300
3000
426
Image
32

---

Published in Transactions on Machine Learning Research (11/2025)
E.2 Proposed, non z-normalized, datasets
Table New datasets collection: datasets from both the UCR archive (dashed line) and the Monash UEA extrinsic regression archive. When missing values and/or varying lengths, replace missing values with 0 and pad series to maximum length with O. All of the datasets aggregated over time step window of length 5, c.g. in the first line, mean test AUC gets 11% better when using [40%, .... 60%] and 16% better with [75%, ... 100%] compared to [ of the scries. Italic datasets are not included when classes are imbalanced as problems become too difficult for the chosen classifiers.
not z-normalized. AUC gain is mean improvement,
Size
Length
Class
Type
AUC Gain AUC Gain test (half/full)
train (half/full)
Data
BME
180
128
Simulated
(7%/7%)
(11%/16%)
Chinatown
365
24
Traffic
(1%/1%)
(0%/0%)
Crop
24000
46
24
Image
(9%/10%%)
(8%/9%)
DodgerLoopDay
158
288
7
Sensor
(4%//5%)
(14%/17%)
BOGVerticalSignal
724
1250
12
EOG
(35%/35%)
(43%/45%)
GestureMMiAirrD
338
360
26
Trajectory
(11%/129%)
(23%/26%))
GunPointAgeSpau
451
150
Motion
(7%/7%)
(7%/7%)
House
135
3000
2
Device
(1%/1%)
(5%/6%)
MelbournePeddstrrian
3650
24
10
Traffic
(15%/159%)
(15%/16%)
PLAID
1074
Vary
11
Device
(0%//0%)
(2%/2%)
Rock
70
2844
Spectrum (1%/1%)
(12%/12%)
SemgHaandGenderCh2
900
1500
Spectrum
(1%/2%)
(11%/13%)
SmoothSubspace
300
15
Simulated
(21%/37%%
(20%/38%)
UMD
180
150
3
Simulated
(10%/119)
(22%/28%)
AcousticContamiiationnnn
138
365
2
Environment (1%/2%)
(5%/7%)
ComminiumCCncentraaatit
629
2542
Environment
(3%/4%)
(5%/10%)
BitcoinSeentiment
332
24
Sentiment
(11%/13%)
(-5%/-6%)
Chilled Chilled Watterreeietorotoo
459
168
Energy
(0%/00%)
(8%/11%)
Copper Concentration
629
2542
Environment
(4%/5%)
(4%/3%)
Covid Andalusia
204
91
2
Health (7%/8%)
(7%/17%)
OiilyOil Prices
188
30
Economy
(25%//23%)
(10%/8%)
DhalaulounlyAAiiuuulity
2068
24
Environment
(2%/3%%
(0%/2%)
MlectricitPPredicee
810
168
Energy
(4%/5%)
(7%/13%)
FloodModeling3
613
266
Environment
(20%/22%)
(20%/31%)
HoushholPPoverrCnnsmpplll
1431
1440
Energy
(4%/7%)
(14%/27%)
PlotwattePrredicoor
351
168
Energy
(1%//1%)
(4%/5%)
MadridPMM10Quuality
6923
168
Environment
(4%/5%)
(11%/16%)
ParkingBBrmmingham
1888
14
Environment
(7%/28%)
(5%/22%)
PrecipitatioonAAndaluiiaa
672
365
Environment
(0%/1%)
(3%/4%)
Siera.NewddaMMoutaaaSSnoo
500
30
Environment
(6%/7%)
(-2%/4%)
Solar DolarRaddiationnAndalss
672
365
Energy
(1%/1%)
(4%/4%)
SteamPredictor
300
168
2
Energy
(2%/2%%
(3%/6%)
Coussnion
364
144
Energy
(5%/5%)
(1%/6%)
WindTurbinePower
852
144
Energy
(11%/12%)
(15%/21%)
33

---

Published in Transactions on Machine Learning Research (11/2025)
Supplementary results
F.1
Additional figures: Standard cost setting
alap
High delay cost
asap
calimera
3
eodire
meatt raik
5
economy
6-
ects
proba threshold
stopping_rs rule
teaser avg cost
9-
teaser hm
10
cnst-informed
11
Low delay cost.
cost-uninformed
12
0.0
0.1
0.2
0.3
0.4
0.3
0.6
0.7
0.8
0.9
1.0
alpha
Figure Standard cost setting, non z-normalized proposed datasets (orange cylinder). Compared to Figure 3a. the global ranking not altered much. One can observe that for [0.5, 0.7] the top group now more populated, gathering the first six approaches, probably due to the limited amount of dalasets available in this case.
F.2
Additional figures: Anomaly detection cost setting
alap
High delay cost
2
asap
calimera
3-
eodire
4-
ecee:
tuea rank
cconomy
ccts
proba_ threshold
7-
stopping_ rule
8-
teaser avg cost teaser hi
9
10
cost-informed cost-uninformed
Low delay cost
0.0
alpha
Figure 11: Anomaly detection cost setting, original UCR datasets (blue cylinder). Compared to Figure 6a see that CALIMERA is now clearly dominating all other methods for all a. The global ranking remains globally stable otherwise.
34

---

Published in Transactions on Machine Learning Research (11/2025)
F.3
Removing calibration
alsp
High delay cost
asup
calimera
3-
codire
4-
co0e
mncat ronk
5
economy
6-
ects
proha threshold
stopping rule
teaser avg cost
teaser hm
10
cost-informed
11
Low delay cost
cost-uninformed
12
0.0
0.1
0.2
0.3
0.4
0.5
0.5
0.7
0.8
0.9
1.0
alpha
Figure 12: Standard cost setting, original UCR datasets (blue cylinder). The calibration step is now removed, i.e. the outputs from the decision function is now simply passed through softmar function. Both CALIMERA and PRoBa THRESHOLD suffer heavily from using uncalibrated scores.
proba threshold no calib
teaser avg cost no calib
stopping rule no calib
teaser hm no calib
calimera no calib
economy no calib
ecdire no callb
ecec no callb
ects no calib
no calib is better
Mean difference
0.01
Mean Difference ==c/č=c// c
-0.01 54
-0,0033
0.0034
0.0071
0.0000 18/38/21
-0,0024
0.0081
0.0001
0,0040 45/0/32 0.2378
0.00
28/0/49
43/0/34
29/0/48
38/1/38
23/0/54
38/0/39
33/0/44
Wilcoxon p-value
0.0288
0.7433
0.7166
0.3770
0.6257
0.0183
0.8490
0.6605
p-value 0.05
-0.01
calimera calib
ecdire calib
ecec calib
economy callb
ects calib
proba threshold callb
stopping rule callb
leaser avg cost calib
teaser hm callb
calib is better
Figure 13: Pairwise comparison [lsmail-Fawaz et al. 2023 calibration (calib C) no calibration (no calib C). We select 0.8 as the alpha value where both the naive baselines cross, i.e. where, in average, most of datasets are more challenging. Square colors are indexed on the mean AugCost difference. For example, CALIMERA has a lower mean AuyCost when trained over calibrated scores: it appears in dark blue. The Wilcoxon p-value is equal to 0.0288, which lower than significance lovel equal to 0.05. Thus, CALIMERA statistically under-performs when using uncalibrated scores. This is also the case for the PROBA THRESHOLD method.
F.4 Ablation study on non-myopic trigger models
One of the key component when it comes to trigger functions is the non-myopic ability to anticipate the likely future. In this section, we conduct ablation study over the anticipation- baase property by making those type of approaches myopic. In particular, ECONOMY and CALIMERA will be of interest here. More precisely, the expected
35

---

Published in Transactions on Machine Learning Research (11/2025)
cost horizon has been limit to 1 for ECONOMY. Concerning CALIMERA, the backward cost propagation has been blocked such that the only available information to trigger be the expected cost difference ween current and next. timestamp.
High delay cost
2-
calimera
calimera myopic
meat rank
coonomy
coonomy myopic
Low delay cost
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
alpha
Figure 14: Standard cost setting, z-normalized proposed datasets (orange cylinder). The onticipation-basee property is blocked from non-myopic methods. The myopic counterparts perform equal when dealing with high temporal cost, but are significantly worse when it gets weaker, i.e. for >> 0.5.
F.5
Changing the base classifier
alap
High delay cost
asap
calimera
3-
ecrlire
coec
mees rank
econony
ects
proba threshold
stopping rule
teaser svg cost
leaser hm
10
cost-informel cost- unttuuniforedd
Low delay cost
11
12
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
alpha
Figure 15: Standard cost setting, original UCR datasets (blue cylinder). 2023). Results are very close to those exposed in Figure 3a
The base classifier is now WFASE: 2.0
Schäfer & Leser
36

---

Published in Transactions on Machine Learning Research (11/2025)
alap
High delay cost
asap
calimera
eedire
4-
Ittesn rank
economy
6-
ects
proba threshold
7-
stopping, rule
teaser _avg__g_ cost
teaser hm
cost-informedl
Low delay cost
cost-uninformed
12
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
alpha
Figure 16: Standard cost setting, original UCR datasets (blue cylinder). The base classifier now a pipeline including
features extraction with TSFRESHI (Christ et al. 2018) and classification using XGBoosT (Chen et al. 2015). Results are a bit noisier than those exposed Figure
3a
Table 8: Comparison of the tested classifiers. Percentage representing, for each alpha, the amount of dataset for which each classifier ranked first, averaged over all trigger models and all datasets. Ties are not considered thus, each line may sum less than 1. Best performing classifier is underlined.
classifier
MINIROCKET
WEASEL 2.0
TSPRESH&&GBOOST
12.60%
11.95%
16.36%
0.1
31.56%
18.44%
37.01%
0.2
32.86%
18.96%
36.49%
0.3
36.49%
19.48%
32.34%
0.4
39.87%
17.53%
31.30%
0.5
43.12%
19.10%
26.62%
0.6
42.60%
23.25%
23.51%
0.7
44.16%
24.94%
20.52%
0.8
48.44%
25.45%
15.71%
0.9
48.70%
26.62%
14.29%
42.21%
26.75%
14.68%
F.6 Impact of z-normalization
Clearly, using z-normalized datasets is not applicable in practice, as it would require knowledge of the entire incoming time series. In research context, previous work has used such training sets to test the proposed algorithms. Our goal here, is to assess whether this could have a large impact on the performances. For example, when normalized time scrics has a low variance at the beginning, we can expect a high variance in the rest of the scries since the mean variance is 1. There therefore an information leakage that can be exploited by an ECTS algorithm, while this not representative of what happens real applications. (2020), where the z-normalization of available time series is repeated each time step, has its own problems. In particular, it means that if single classifier is used for all time steps, the representation of x, can be different at times f and + 1 and all further time steps which can induce confusion for the classifier.
proposal such as the one presented by [Schater & Leser
37

---

Published in Transactions on Machine Learning Research (11/2025)
On the one hand, z-normalization induces an information leakage that could help methods to unduly exploit knowl- edge about the future of incoming time series. On the other hand, any normalization rescales the signal and thorefore, potentially, hinder the recognition of telltale features. So, does z-normalization affect the performance of ECTS methods? And yes, in which way?
In order to answer this question, we took the new datasets collection described in Section 4.1 They are indeed z-normalized originally. We duplicate and z-normalized them to get sccond collection. As explained in Section 4.1 some extrinsic regression datasets have been converted into classification ones. Here, the threshold value chosen to discretize the output into binary classes has been set to the median of the regression target. In this way, classes within those datascts are equally populated.
In these experiments, the delay cost is linear as Section 1.2 and as in most of the literature. Figure 17 reports pairwise comparisons done on the 35 datasets. We look differences observed. One can see that most of the trigger models do not actually benefit from the z-normalization. Quite the opposite: out of nine trigger models, only one, i.c. ECDIRE, actually has better mean AugCost when being trained on z-normalized data. Regarding the remaining methods, both STOPPING RULE and TEASERAny perform significantly worse when operating on z-normalized data. Those trends are quite similar for other a values, without any siguificance on the statistical tests though. Thus, while z-normalization has some impact, since privileged information from the future can be leaked, our experiments, for the proposed datasets collection at least, show that this does alter the overall results reported in the literature, and are globally accordance with the results presented in Section
0.8, as this is the only value for which significant
proba threshold
teaser avg. cost Z
stopping_run rule Z
teaser Z Z
calimera Z
ecanomy Z
ecdire Z
ecec Z
ects Z
Z is better
Mean difference
0.02
Mean Difference << = Z
-0.0130
0.0092
-0.0097
-0.0116
-0.0020
-0.0134
-0.0161
-0.0328
-0,0149 10/0/25
0.00
9/1/25
17/0/18
13/2/20
15/1/19
17/2/16
11/2/22
8/2/25
8/0/27
Wilcoxon p-value
0.2224
0.8652
0.1472
0.8506
0.4762
0.2224
0.0368
0.0091
0.2012
p-value 0.05
-0.02
callmera Z
ecdire Z
ecec Z
economy 2
ects Z
proba threshold Z
stopping rule Ž
teaser avg cost Ž
teaser hm Z
is better
n-ormaliization (Z) vs no z-normalization (2), = 0.8. Square colors are indexed on the mean AugCost difference. For example, CALIMERA has a lower mean AngCost when trained over non z-normalized datascts by 1.3c-2 and appears in light blue. It beats the z-normalized version over 25 datasets, loses over 9 and are lied on 1. The Wilcoxon p-value is equal 0.2224, which higher than significance level equal 0.05. Thus, no statistical difference can be observed for the considered approach.
Figure 17: Pairwise comparison [Ismail-Fawer et al. 2023
38

---

Published in Transactions on Machine Learning Research (11/2025)
proba threshold
teaser avg. cost
stopping rule
teaser hm
economy
ecdire
ecec
ects
0.10
non z-normalized
0.0813
0 0.0429
+0,0068
0.0796
-00074
-001300
0.0307 -0,0177
read by lines
29/1/5
26/1/8
15/1/19
33/1/1
12//22
8/126
19/0/16
22/0/13
calimera
1e-04
0.0004
0.1103
1e-04
0.1103
0.0023
0.9649
0.0210
z-normalized
0..5
Mean difference
read by columns
-0.0384
-0 0.0880
-000017
-0 0.0887
+0 -0.0942
-0. -0.0829
-0.0506
10/4/21
2/1/32
16/1/18
2/1/32
1/1/33
6/0/29
11/0/24
ecdire 0.0087
0.0134
1e-04
0.8763
1e-04
1e-04
1e-04
0..00
0.0590
-0 10.0497
0.0367
-0. .0.0503
-0.0658
-0 00,0446
-0.0122
ecdire
30/0/5
ecec 0.7158
1e-04
1e-04
0.0055
0.0001
< 1e-04
0.0074
-0.05
0.0396
-0.0195
0.0864
-0, 40,0006
-0.0062
0.0051
0.0375
ecec
23/1/11
14/2/19
31/1/3
16/1/18
13/1/21
21/0/14
24/0/11
economy
0,0090
0.1103
1e-04
0.9022
0.1819
0.4128
0.0014
-0.10
-0.0083
0.0673
-0,0478
-0,0870
-0 0.00925
-0,0812
-0.0489
economy
12/1/22
5/0/30
8/1/26
1/1/33
2/1/32
5/0/30
10/0/25
ects
1e-04
0.2224
0.0025
1e-04
1e-04
1e-04
0.0032
0.0685
0.0095
0 0.0289
0.0768
-0.0056
0.0057
0.0381
ects
29/1/5
15/0/20
22/1/12
29/155
14/3/18
22/0/13
24/0/11
proba threshold
0.0579
0.0230
0.2350
1e-04
0.3266
0.3678
0.0032
-0.0071
-0.0661
-0.0466
0.0012
-0.0756
0.0113
0.0437
proba_ toldsol
15/1/19
4/0/31
9/1/25
20/2/13
4/1/30
25/0/10
27/0/8
stopping_rule
0.9163
1e-04
1e-04
0.0021
0.0371
0.0275
1e-04
-0.0099
-0.0690
-0.0495
-0.0017
-0.0785
20,022
0.0324
stopping_ruke rule
8/1/26
3/0/32
7/1/27
19/2/14
3/1/31
12/4/19
20/11/4
teaser teaser_avg__on
0,0012
0.0543
0.0011
< 1e-04
0.3592
0.9935
0.0004
0.0181
0.0400
-0.0215
0.0264
-0,0504
0.0252
0.0281
teaser_avg.r avs__costtttt___
22/0/13
12/0/23
19/0/16
30/0/5
9/0/26
28/0/77
29/0/6
Mean Difference <<c/r= c/r==//>e
1e-04
0.2833
0.6642
1e-04
1e-04
1e-04
1e-04
c<r/r= c/c
0.0325
-0 00.0265
-0.0070
0. 0.0408
-0 -0.0360
0.0396
0.0425
0.0144
Wilcoxon p-value
teaser hm
22/0/13
11/0/24
20/0/15
33/0/2
10/0/25
27/0/8
29/0/6
14/11/10
p-value 0.05
0.0004
0.1276
0.0003
0.0002
0.2056
calimera
ecdire
ecec
economy
proba threshold
ects
stopping rule
_avg_cas_s cost
teaser avg
[ssmailFawaz et al. 2023 displays the comparison of the competitive methods trained over non z-normalized dataset (orange cylinder). The values within this triangle has to be read by lines, i.e. for a considered line, red shades indicate better performances, blue shades weaker performances. The lower triangle, with dark red contours, is the comparison the methods trained over the same datasets, z-normalized (chocolate cylinder). The values within this triangle has to be read by columns, ie. for considered column, red shades indicate better performances, blue shades weaker performances. The complete figure being symmetrical indicates that z-normalization does not impact much relative ranking bet methods.
Figure 18:
The upper triangle, with dark blue contours,
39

---

Published in Transactions on Machine Learning Research (11/2025)
F.7
Anomaly detection cost setting an ablation study
Exponential delay cost only
alap
High delay cost
2-
asap
calimera
ecdlire
ecec
meat ratik
economy
ects
proba threshold
stopping rule
teaser avg cost aosto
teaser hm
10
cost informed
11
Low delay cost
cost unin-friffree
12
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
alpha
Figure 19: Exponential delay cost, symmetric binary misclassificatiin cost, non z-normalized proposed imbalanced datasets (orange cylinder with whole).
Imbalanced misclassification cost only
alap
High delay cost
asap
calimera
3-
ecrlire
coec
meees rank
econony
ects
proba threshold
stopping rule
teaser svg cost
Leaser hmm
10
cost-informed cost- cost-unintee uninffrrood
Low delay cost
11
12
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
alpha
Figure 20: Linear delay cost, non symmetric imbalanced misclassification cost, non z-normalized proposed imbalanced datasots (orange cylinder with whole).
40

---

Published in Transactions on Machine Learning Research (11/2025)
Standard cost setting, imbalanced datasets
alap
High delay cost
asap
calimers
eedire
ecee
ttean ranks
economy
eets
proba threshold
71
stopping_ rule
teaser aos_ cost
teaser
10
cost-informed
11
Low delay cost
cost-uninformed
12
0.2
0.4
0.5
0.7
0.9
1.0
alpha
Figure 21: Linear delay cost, symmotric binary misclassification cost, non z-normalized proposed imbalanced datascts (orange cylinder with a whole).
41


Reviews:
[{"review_id": "r0OBYyvlYm", "reviewer": "Reviewer_zkBX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The proposed taxonomy appears to be novel and categorizes ECTS approaches in reasonable classes.\n- The paper proposes a new simple baseline for ECTS.\n- The visualizations presented in Figure 5 and Figure 8 intuitively illustrate the tradeoff between the two costs.\n\n# Weaknesses\n\n- A proper discussion of related work is missing. Therefore, it is difficult to assess the novelty of the proposed taxonomy and the evaluations.\n- The paper is very opinionated and does not provide sufficient evidence for all its claims. For example, the claims about related work and their use of baselines, evaluation protocols, or datasets, or when the paper argues for the proposed evaluation protocol.\n- The paper is not well-written.\n  - The paper is repetitive. For example, the first two paragraphs of Section 3.1 repeat individual sentences and entire explanations detailed in the prior section. Another example are the cost functions, which are introduced multiple times throughout the paper.\n  - The paper is inconsistent in its use of terminology and formalism. For example, some elements or notations are not properly defined, such as ECTS functions, and some minor aspects have dedicated equations, for example eq. (6) and (7), while the definition of a separable model does not. \n  - The formatting is inconsistent. For example, the alignment of the figures.\n  - The paper is often imprecise inviting additional questions. For example, is the length of each time series $T$ a global parameter of the dataset? If not, is there not also some temporal leakage for methods using this information?\n- The evaluations are limited.\n  - The evaluations compare 11 decision models of separable approaches. However, based on Table 1 there seem to be more methods than the ones evaluated here. Especially noticeable is the omission of more recent methods.\n  - The datasets are selected based on their ability to be classified by one particular algorithm. Does this procedure not introduce selection bias? Additionally, it is not explained how the selection criteria are computed. The paper suggests the use of the proposed evaluation protocol for future evaluations. Does this apply to non-separable approaches? Has the second selection criteria been evaluated for the normalized datasets? \n  - If only the decision functions are of interest for the evaluations, could we use synthetic classifiers instead?\n  - The paper only presents aggregated results, which makes them of limited use for future evaluations and direct comparisons.\n- There are no details on the proposed library.\n- The survey of methods does not place each method properly in the proposed taxonomy and is generally inconsistent in the level of detail. Additionally, the survey appears incomplete, as other surveys consider more methods than the ones discussed here. \n\n# Minor Comments\n\n- It is a bit odd to call the second setting of the evaluation \"anomaly detection\", which is in itself a well-studied field that, especially on time series, significantly deviates from the setting considered here. \n- Figures 1 and 2 are difficult to parse.\n- Why do alap and asap not appear in the ranking evaluations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses methods for early classification in time series (ECTS). It introduces several criteria by which one can classify ECTS methods. The paper distinguishes methods that consist of a separate classification and decision component, and methods that work end-to-end. The paper classifies separable approaches based on whether they use the cost during training, whether they use the cost during inference, how much context is used in the decision component, whether the model relies on predictions, and how many classifiers are used. The paper classifies 20 methods with these criteria. Furthermore, the paper curates a collection of datasets for benchmarking ECTS methods and presents two experiments where 11 methods are compared including a novel method introduced in this paper. Additional experiments are provided in the appendix.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In my opinion the paper could be significantly improved by shortening it considerably (maybe 12 pages would be enough), removing repetition, and increasing precision and clarity. In my opinion, the paper should focus on the taxonomy and the evaluation and less on the survey. Section 3 could be moved to the appendix, where it could be extended to provide more details on each method, in particular, in the context of the proposed taxonomy. Most importantly, the paper should provide convincing evidence and citations for all claims, including a thorough discussion of related work. The evaluations could be improved with more methods and a more precise discussion of the setup of the evaluation protocol, in particular, regarding the selection of methods and evaluation criteria. Lastly, the conclusion could be improved with a more explicit discussion of the results and the impact of the presented evaluations. In the current version, the insights from the evaluations do not seem to match the main experiments or the goals of the evaluation. (For more details, see the discussion of weaknesses.)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ARSD7kFA4", "reviewer": "Reviewer_sR8Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n*The proposed taxonomy is not arbitrary but rooted in ECTS’s core challenges, ensuring it captures the essence of different methods. The evaluation protocol further enhances rigor by standardizing key variables.\n\n*The paper prioritizes scenarios that matter for industry and healthcare. This practical focus ensures the paper’s findings can be directly applied to solve industrial problems.\n\n*The open-source library and detailed experimental documentation set a new standard for ECTS research. The library includes full code. The paper also reports supplementary results in appendices, leaving no gaps in the evaluation.\n\n**Weaknesses**\n\n*The paper explicitly excludes end-to-end methods from experiments, stating they are “left for future work.” This is a major limitation, as end-to-end methods which integrate classification and triggering into a single model, often using deep learning have gained traction in recent years. By ignoring these methods, the benchmark fails to provide a complete picture of state-of-the-art ECTS performance. This omission also limits the paper’s utility for researchers working on deep learning-based ECTS, who cannot compare their methods to the benchmarked separable approaches.\n\n*While the paper introduces 34 non-z-normalized datasets, this collection has two flaws. First, it is small compared to the 77 UCR datasets used in standard settings, reducing the statistical power of experiments on anomaly detection. Second, the datasets lack diversity in key characteristics: most are from energy or environment domains, with few from high-stakes fields like healthcare or finance. This limits the generalizability of the paper’s findings. Additionally, the paper converts some regression datasets to classification by thresholding, but it does not validate whether these converted datasets reflect real-world classification tasks.\n\n*The paper tests two cost settings (balanced/linear and unbalanced/exponential) but ignores more realistic, complex cost functions common in practice. In healthcare, delay cost may depend on the true class, or misclassification cost may be stochastic. The paper only mentions these scenarios as “future work” but provides no analysis of how existing methods would adapt. This limitation reduces the paper’s relevance to real-world applications, where cost functions are rarely static or independent of class.\n\n*While the paper conducts ablation studies on classifier calibration and base classifiers, it does not analyze the impact of individual components within trigger functions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work makes an impactful contribution to the early classification of time series field from a perspective of survey paper. \n\n(1) It proposes a principle-based taxonomy for early classification of time series methods that systematically organizes existing approaches by four key dimensions. \n\n(2) It conducts extensive, rigorous experiments to benchmark 9 state-of-the-art early classification of time series algorithms across diverse scenarios, addressing past evaluation flaws.\n \n(3) It develops an open-source library that implements most existing algorithms  and integrates the proposed evaluation protocol. \n\n(4) It introduces a collection of 34 non-z-normalized datasets to address information leakage in traditional evaluations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*The paper should extend its experiments to include 3–5 representative end-to-end methods to provide a complete state-of-the-art comparison.\n\n*To address the limited diversity and scale of non-z-normalized datasets, the paper is suggested to: (1) Add new datasets from high-stakes domains to increase domain coverage. (2) Include datasets with varied characteristics to test method robustness. (3) Validate converted regression datasets by consulting domain experts.\n\n*To fill the gap in component-level analysis, the paper is suggested to perform ablation studies on the key components of 3–4 benchmarked trigger functions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OZusnTPnb8", "reviewer": "Reviewer_evND", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nS1.\nThe proposed characterization of components of ECTS methods may be natural. The previous work (Gupta et al., 2020) categorized ECTS methods into prefix based, shapelet based, model based, and miscellaneous approaches. I feel that the proposed characterization is more useful than that of (Gupta et al., 2020).\n\nS2.\nExperimental comparisons are clearly purposeful, sufficiently large, and reliable.\n\n___\n\nWeaknesses\n\n(I could not find serious weaknesses in the 1st review, and the followings are minor weaknesses.)\n\nW1.\nThe authors define the training data as $(\\\\mathbf{x}\\_T^i,y^i)\\_{i\\\\in[0,M]}$, which is a gross simplification (Sec.1, Para.3, P.1). For example, the authors described \"The number of measurements, and hence the input dimension, varies.\" (Sec.2.4, Para.1, P.8), but this assertion does not fit with the simple definition of the training data. It's not that the assertion is hard to understand, but the writing style is a bit sloppy. They will need to either clearly generalize $T$ to $T\\_i\\in[0,T]$ or describe some kind of excuse.\n\nW2.\nThere is a discrepancy in the notation for $\\\\mathcal{L}$: $\\\\mathcal{L}(\\\\cdot,y,t)$ above Equation (1) and $\\\\mathcal{L}(\\\\cdot,t,y)$ in Eq.(1).\n\nW3.\nEq.(3) needs to include the randomness of $y$.\n\nW4.\nFigure 1 and Table 1 should be written in more detail. For example, they should include all the keywords used to characterize the trigger function such as \"blind\" or not, \"confidence-based\" (\"instance-based\" or \"sequence-based\") or not, \"cost-informed\" (\"at training\" or \"at test\") or not (\"cost-uninformed\"), and \"anticipation-based\" or not (\"myopic\"). Looking at Table 1, it is not clear whether \"Confidence ✓\" is \"instance-based\" or \"sequence-based.\" Also, in Figure 1, can you use a tree structure to characterize the trigger function?\n\nW5.\nThe authors write only \"AvgCost is the most appropriate criterion to both train and evaluate ECTS approaches.\" under Eq.(4), while they write \"We claim that the AvgCost is the appropriate measure by which to evaluate the performance of the methods. This is indeed what will be \"paid\" at the end of the day by a practitioner using a method.\" (Sec.5, Para.1, P.25). The authors should also describe a discussion of the appropriateness of AveCost under (4).\n\nW6.\n\"which we call episodes\" loses the comma (Sec.3.3, Para.2, P.13).\n\nW7.\nThe authors should define \"noisy cost\" and \"cost drift\" or explain those with reference (Sec.5, P.26).\n\nW8.\nWhat methods do the authors envision for exploiting the misclassification cost in training? Is it something like the following?\n- Pires, B. A., Szepesvari, C., & Ghavamzadeh, M. (2013, May). Cost-sensitive multiclass classification risk bounds. In International Conference on Machine Learning (pp. 1391-1399). PMLR.\n- Charoenphakdee, N., Cui, Z., Zhang, Y., & Sugiyama, M. (2021, July). Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning (pp. 1507-1517). PMLR.\n\nI think that such a topic is relevant to general classification methods and not specific to ECTS, so it may be redundant. If the authors believe that a discussion specific to ECTS is possible, authors should provide more details.\n\nW9.\nFormatting Instructions for TMLR Journal Submissions (Latex template) writes \"The table number and title always appear before the table\". Change the location of captions for Tables 1-5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses the problem, \"early classification of time series (ECTS)\", to construct a model that predicts a class label as early as possible with a smaller misclassification cost, based on repeated sequential observations of a certain phenomenon. The authors are concerned about a lack of a systematic and shared evaluation protocol to compare the relative merits of various methods.\n\nTherefore, the authors first listed and organized general components of ECTS methods, to clarify the differences between various methods, as follows:\nECTS methods can be end-to-end or separable, and separable ECTS methods consist of a trigger function and a classification function.\nA trigger function is characterized by either \"cost-informed\" (\"at training\" or \"at test\") or not (\"cost-uninformed\"), and either \"anticipation-based\", \"myopic\", or \"blind\" that is determined independently of learning, and a cost-uninformed trigger function is categorized into either \"confidence-based\" (either \"instance-based\" or \"sequence-based\") or \"blind\".\nA classification function is characterized by whether it is \"at each timestamp (classifier collection)\" or \"projection-based to treat all timestamps uniformly (single classifier)\", in addition to the choice of base classifier such as SVM or NN.\n\nThe authors then discussed the significance of each characteristic of the ECTS method based on extensive experimental comparisons of various ECTS methods using many datasets. The findings are summarized in Sections 4.2.2, 4.3.2, 4.4, and 5, and the program code used will also be published.\n\nThis paper is a survey, and the contributions of this paper are\n- characterization of components of ECTS methods,\n- extensive experimental comparisons\n\nto encourage future systematic development.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please change regarding W1,2,3,6,7,9.\nThis is critical to securing my recommendation for acceptance.\n\nPlease make changes the authors think would be beneficial regarding W4,5,8.\nHowever, I would like you to consider revising for W4 strongly.\nThese would simply strengthen the work in my view.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)