# AutoTiM - An Open-Source Service for Automated Provisioning and Operation of Time Series Based Machine Learning Models

Andre Ebert, Jakob Kempter, Marina Siebold, Robert Pesch, Tetyana Turiy, Tevin Tchuinkam, Thomas Caffin Sune

<figure><img src="image_1.png" /></figure>

## To cite this version:

Andre Ebert, Jakob Kempter, Marina Siebold, Robert Pesch, Tetyana Turiy, et al. AutoTiM - An Open-Source Service for Automated Provisioning and Operation of Time Series Based Machine Learning Models. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.266-278, 10.1007/978-3-031-34107-6_21. hal-05498094

HAL Id: hal-05498094

https://inria.hal.science/hal-05498094v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# AutoTiM - An Open-Source Service for Automated Provisioning and Operation of Time Series Based Machine Learning Models

Andre Ebert(✉)D, Jakob KempterD, Marina SieboldD, Robert Pesch, Tetyana TuriyD, Tevin TchuinkamD, and Thomas Caffin SuneD

inovex, Karlsruher Straße 71, 75179 Pforzheim, Germany
{andre.ebert,jakob.kempter,marina.siebold,robert.pesch,tetyana.turiy,
tevin.tchuinkam,thomas.caffin-sune}@inovex.de
https://www.inovex.de/

**Abstract.** The ubiquitous availability of heterogeneous sensor data created by *Internet-of-Things* (IoT) technologies and *Industry 4.0* trends drastically accelerated the development of machine learning applications. *AutoML* services enable users with sparse machine learning knowledge to develop AI-based applications and rapidly evaluate the feasibility of data-driven ideas. Therefore, there exists a demand for holistic, low-code, end-to-end *AutoML* systems, which cover all stages of the machine learning lifecycle (i.e., feature engineering, model training, evaluation, versioning, provisioning, etc.). Although there are proprietary, cost-intensive platforms addressing these issues, no open-source solutions covering these aspects are known to us. In this paper we present *AutoTiM*, an open-source service capable of creating and operating highly performant machine learning models without requiring domain expertise or machine learning knowledge.

**Keywords:** AutoML · Democratization · Automated Machine Learning

## 1 Introduction

IoT technologies offer vast opportunities to gather information from various data sources and to aggregate it within data lakes or lakehouses [8]. This accelerates the adoption of data-driven applications for consumer use as well as for industrial purposes, such as predictive maintenance, context-aware services, energy consumption forecasts, etc. Machine learning models are usually at the core of such applications, therefore, *Automated Machine Learning* (AutoML) libraries and tools have received a lot of attention lately. *AutoML* enables automated

Supported by Service-Meister, a research project funded by the german Federal Ministry for Economics and Climate Action (BMWK), https://www.servicemeister.org/.
model training and evaluation without requiring domain expertise; it is also a
critical component of holistic *Machine Learning Operations* (MLOps) architec-
ture [18]. While big companies like *Google* already provide comparatively mature
*AutoML* solutions, the number of such holistic, end-to-end systems available as
open-source software is rather sparse. Most of free-of-charge libraries focus on
specific aspects of machine learning like feature engineering or model training
individually, rather than combining all necessary steps to create and operate
machine learning models within one system. Hence, a holistic and publicly avail-
able end-to-end *AutoML* service would offer great value for various stakeholders,
be it machine learning experts or novice users. Furthermore, it would eliminate
the need to transfer complex domain expertise through a time-consuming process
of data exploration, feature engineering, model training and evaluation. Small
and medium-sized companies with limited machine learning expertise would be
empowered to independently deploy intelligent applications. Thus, the holistic
approach to *AutoML* encompassing all of these aspects is a crucial precondition
for the democratization of machine learning.

In the following, we provide an overview of related work in Sect. 2. After-
wards, we present *AutoTiM*, an end-to-end and easy-to-use *AutoML* open-source
service in Sect. 3. The evaluation in Sect. 4 demonstrates *AutoTiM's* capabili-
ties to provide highly competitive machine learning models without undertaking
any manual steps and without the inclusion of domain expertise. Finally, we
summarize our findings and outline future work in Sect. 5.

2 Related Work

*AutoTiM* is a low-code, end-to-end *AutoML* service focused on time series classi-
fication. In the following, we define time series classification and look at specific
application fields in industry, namely, predictive maintenance or anomaly detec-
tion. Moreover, we provide an overview of the existing *AutoML* approaches.

## 2.1 Time Series Classification and Its Application

*Time series* are collections of chronologically organized data obtained from
sequential measurements [6], whereby data sources can be industrial sensors,
medical monitoring devices, and others. A decade ago, its analysis with algo-
rithms of supervised classification was perceived to be a challenging task, while
today there are numerous productive applications in the areas of *IoT* and *Indus-
try 4.0* as well as in the consumer market [13, 14].

A characteristic example for the application of time series classification in
production is *Predictive Maintenance* (PdM), which is a maintenance manage-
ment strategy resulting from its predecessor strategies 1) *Reactive Maintenance*
and 2) *Preventive Maintenance*. By employing predictive tools as well as algo-
rithms of machine learning, continuous monitoring of production processes and
machinery is implemented. Goals are the optimization of maintenance costs,
the minimization of downtimes, and increasing productivity [3]. Naturally, the
effectiveness of any *Predictive Maintenance* concept relies on the algorithmic approach it is based on, i.e., machine learning models, statistical approaches, or mathematical models [9]. Currently and especially when not automated, the efficient creation of prediction models by manual integration of algorithms requires a lot of domain expertise for feature engineering, selection of algorithms, training of models, etc. [10]. An example illustrating these challenges during segmentation, classification, and anomaly detection with sensor data is given by Coelho et al. [4]. Other examples within the automotive and the railway industry are given by the authors of [5].

*Anomaly Detection* is an important field for *Predictive Maintenance* as it deals with identifying outliers that deviate from an expected distribution [16]. Though it could be interpreted as a binary supervised classification problem (*anomaly* vs. *no anomaly*) the sheer application of supervised algorithms is not sufficient in reality. Reasons among others are labor-intensive labeling and common strong class imbalances in datasets [16]. Hence, the application of unsupervised, self- and semi-supervised toolkits for *Anomaly Detection* covering conventional as well as *Deep Learning* based concepts gains attention [2, 12].

<table><caption>**Table 1.** Machine learning automation levels as presented in [10].</caption><thead><tr><th></th><th>Systems</th><th>TF</th><th>PE</th><th>FE</th><th>ML</th><th>ATV</th><th>RSR</th></tr></thead><tbody><tr><td>Level 6</td><td>Fully automated ML agent</td><td>?</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Level 5</td><td></td><td>ComposeML + Level 4 systems</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Level 4</td><td></td><td>IBM AutoAI, AutoBazaar, RapidMiner, AutoTiM</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Level 3</td><td>ML + ATV combined</td><td>Rafiki, Amazon SageMaker, H2O, ATM, AUTO-WEKA</td><td></td><td></td><td>✓</td><td>✓</td><td></td></tr><tr><td>Level 2</td><td>ML + ATV separate</td><td>scikit-learn, Keras, ORANGE, Tensorflow, Pytorch, WEKA</td><td></td><td></td><td>✓</td><td>✓</td><td></td></tr><tr><td>Level 1</td><td></td><td>Implementation of Decision Tree, KMeans, SVM, etc.</td><td></td><td></td><td>✓</td><td></td><td></td></tr><tr><td>Level 0</td><td>No automation</td><td>Programming languages (python, C++, etc.)</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

## 2.2 Automated Machine Learning

Commonly, *AutoML* deals with the automation of one or more steps of the *Machine Learning Lifecycle* due to its iterative, labor-intensive nature. Karmaer et al. define 6 levels of maturity for *AutoML* systems based on the number of tasks handled without user interaction [10]. These are *Task Formulation (TF)*, *Data Visualization, Cleaning, and Curation (DCC)*, *Prediction Engineering (PE)*, *Feature Engineering (FE)*, *Machine Learning Model Development (ML)*, *Alternative Model Exploration, Testing, and Validation (ATV)*, and *Result Summary and Recommendation (RSR)* (see Table 1). Within this taxonomy, *AutoTiM* is classified as Level 4, which includes automated feature engineering, model development, ATV, model provisioning, and operation.
In contrast to the above-named proprietary and potentially costly services by IBM<fnref n="1" />, Amazon<fnref n="2" />, and Google<fnref n="3" />, AutoTiM is fully open-source. The partially open-source platform RapidMiner<fnref n="4" /> and the fully open-source framework Auto-Bazaar support a wider range of data inputs and machine learning subtasks [17]. Other than AutoTiM, no other Level 4 system available as open-source software provides comparable capabilities of automated versioning, evaluation, and deployment of machine learning models.

# 3 Concept

This Section examines core requirements for AutoTiM and provides an overview of its architecture. Extensive documentation of all functionalities is also provided at the official Github repository: https://github.com/inovex/AutoTiM.

## 3.1 Requirements

We define the following requirements for holistic ready-to-use AutoML services:

1. **End-to-End** Provide all functionality necessary to train, evaluate, version, and serve machine learning models for a generic, time-series-based use-case.

2. **Environment Independence** Local operation for rapid prototyping as well as operation in a cloud-based production infrastructure is supported.

3. **Simplicity** Interfaces are standardized and easy to understand.

4. **No domain knowledge** is needed for feature extraction and model training.

5. **Adaptability** to various domains, data sources, and use-cases without restrictions as long as the input data is time-series-based.

6. **Model Quality** Performance has to be competitive in comparison to manual machine learning approaches.

## 3.2 Technical Notes

AutoTiM is built with the use of several open-source technologies; in the following, specific design choices are illustrated. All service components depicted in Fig. 1 are deployed as multi-container Docker applications that communicate via REST APIs. For feature engineering, model exploration, training, and evaluation, existing libraries including `tsfresh`, `H2O AutoML`, and `scikit-learn` were applied. Recent evaluations show that none of the currently existing AutoML libraries significantly outperforms the others [7, 19]. AutoTiM makes use of H2O's model training and exploration capabilities, which provide a wide range of algorithms including Deep Learning. These are neither present for `auto-sklearn` nor for `mljar-supervised` while for `TPOT` support of Deep Learning is only experimental. Auto-Keras, on the

<footnote n="1">¹ https://www.ibm.com/cloud/watson-studio/autoai.</footnote>

<footnote n="2">² https://aws.amazon.com/machine-learning/automl/.</footnote>

<footnote n="3">³ https://cloud.google.com/automl.</footnote>

<footnote n="4">⁴ https://rapidminer.com/platform/.</footnote>
other hand, only supports Deep Learning. Furthermore, compared to frameworks like Auto-Gluon or MLBox, H2O can be considered a mature framework with an active community of users and developers [7].

For automated extraction of expressive features, AutoTiM employs tsfresh. Unlike comparable solutions like featurewiz, autofeat, or featuretools, it is explicitly designed for the analysis of time series. Other time series focused frameworks like tsflex and tsfel do not offer important functionalities like relevance-based feature selection for classification tasks and are generally considered less mature than tsfresh.

## 3.3 Architectural Overview

Subsequently, the four key components of AutoTiM's architecture are presented. Namely, these are the Filesystem, the Training and Prediction Services, and the MLflow instance for persisting models and related parameters (see Fig. 1).

**Endpoints.** The complete AutoTiM workflow is accessible through three REST endpoints:

- `/store` is used to persist a dataset in the Filesystem and to tag it with a use-case name and a unique dataset identifier. Depending on the user configuration, a cloud storage or a (local) Docker volume can be used.

- `/train` starts an automated feature extraction followed by model training and evaluation (see 3.3 Training Service); models are versioned in MLflow.

- `/predict` generates predictions for a set of raw data points using a previously trained model (see 3.3 Prediction Service).

**Training Service.** The Training Service as depicted in Fig. 1 is responsible for automated feature engineering, model training, model evaluation, and model preservation in MLflow (see 3.3 Tracking and Versioning). To trigger training for a specific use-case, there are two mandatory parameters needed: 1) the use-case identifier, used for tagging and allocating data for a specific use-case, and 2) the dataset identifier. Optional parameters allow the definition of an upper limit for the feature set size, of the duration of training and model exploration as well as of a custom metric to evaluate a model's performance (see also Sect. 4.2). The logical flow within the Training Service is depicted in Fig. 2. The first step is automated feature extraction with tsfresh. Once extracted, features are selected based on their statistical significance for the classification task at hand. The settings used for extracting features also need to be persisted as a model artifact, in order to reuse them for extraction during prediction (see 3.3 Prediction Service). Output of this step is a feature set, which is used to train several prediction models by utilizing H2O AutoML. Thereby, a diversity of base models, including GBMs, Random Forests, Deep Neural Networks, and GLMs are trained, cross-validated, and subjected to hyperparameter tuning [11]. Two additional stacked ensemble models are formed from the base models to obtain the best classification performance. All learners are evaluated further based on their accuracy.
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Overview of the *AutoTiM* architecture, encompassing the `/store`, `/train`, and `/predict` endpoints as well as the *Filesystem*, the *Training* and *Prediction Services* and an *MLflow* instance.</figcaption></figure>

The best-performing model is versioned in *MLflow* and made available for live predictions. Along with the model, all performance metrics, a confusion matrix, feature settings, and additional metadata are stored. The last step is benchmarking against previously trained models for the actual use-case and dataset combination. Several metrics are calculated using *scikit-learn*, including, *Accuracy*, *Balanced Accuracy*, *Recall*, *Precision*, and *F1-score*. Thereby, the performance of a new model is compared to the performances of existing models in *MLflow*. The best model is tagged for production and deployed to be used for prediction generation.

**Prediction Service.** Once a model for the desired use-case has been generated by the *Training Service*, a prediction can be made for new instances by querying the *Prediction Service*. Three parameters are required: 1) the use-case identifier used for tagging and allocating a set of datasets for a specific use-case, 2) the dataset identifier itself, and 3) one or more time series instances to be classified.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Flow chart of the *Training Service*, encompassing the acquisition of data, the automated extraction features, the training of prediction models, and their evaluation including the selection of the best-performing model and its tagging for production.</figcaption></figure>

The instances must have the same dimensions as the dataset uploaded for training. For prediction the model tagged for production is used per default. A specific version can be requested explicitly by providing an optional `/predict` endpoint parameter. When triggering the *Prediction Service*, suitable feature settings and the model itself are retrieved from *MLflow*. The feature settings are used to calculate a feature set with the same specifications from the given time series instances as it was calculated during the prior training. The *Prediction Service* is capable of classifying single as well as multiple instances and returns a list with predicted labels.

**Result Tracking and Model Versioning.** Versioning and benchmarking of machine learning models, persisting their corresponding parameters and metrics as well as the provision of models for production can be organized by different platforms, such as *MLflow*, *Weights & Biases*, or *Neptune*. Due to its widespread community support, its free availability, and its matured condition, *MLflow* is used to fulfill these requirements for *AutoTiM*. Upon the first launch of *AutoTiM*, a local *MLflow* instance is created and made accessible for the service in order to log experiments, models, and their parameters for each use-case, respectively. For each combination of use-case and dataset, multiple models can be stored. Logged parameters encompass 1) training parameters, 2) performance metrics (see 3.3 *Training Service*), and 3) further artifacts: a confusion matrix, feature settings, model weights, etc.
# 4 Evaluation

To demonstrate AutoTiM's capabilities regarding the classification of time series as well as regarding the ability to generalize across heterogeneous use-cases and domains of application, it was evaluated on the UEA MTSC archive [1]. To verify the quality of models generated with AutoTiM, we compared its results to 20 other recent approaches examined in the Great multivariate time series classification bake off [15]. In the following, we first introduce the datasets as well as the methodology used for evaluation. Secondly, we present and discuss the evaluated results for AutoTiM.

## 4.1 Datasets

The UEA MTSC archive used for evaluation consists of 30 datasets covering a variety of domains [1]. Thereof, 26 datasets contain time series instances of equal length: 8 datasets of electrical biosignals (electrocardiograms, electroencephalogram, and magnetoencephalography), 7 of accelerometer data, 3 of coordinates in Cartesian space, 3 of audio data in spectrogram format, and 5 containing miscellaneous data sources (see Table 2). Besides the data itself, benchmarkable results and documentation are available at the corresponding website$^5$. Table 2 provides an overview of the archive datasets and their specifications. Varying dataset attributes, i.e. dimensions, time series length, etc., are indicating the holistic approach AutoTiM was aiming for. Out of 26 datasets in Table 2, 20 were included in the evaluation. Reasons for leaving out six sets were: 1) the maximum time given for feature engineering was exceeded (4 sets), and 2) deviating data format (2 sets).

## 4.2 Methodology

In order to simplify dataset preprocessing, missing values are filled with a built-in naïve data imputation strategy: backward fill followed by forward fill, whereby each dimension is handled independently from others. For evaluation, the number of extracted features is limited to 1000 in order to address technical preconditions such as limited resources (memory, processing power). Three experiments were conducted with fixed training times of 2, 10, and 30 min for each dataset in Table 2, respectively. During the 2 min of training time, 10 models are generated for each dataset; for all other time spans, 5 models were generated. The automated feature extraction and selection as well as model training and tuning are performed for each run. A fourth experiment was conducted with a fixed maximum number of models to train $n_m = 10$ and a training time $t_{train}$ which was calculated dynamically with respect to the number of features $f$ and dataset size $d$: $t_{train} = \min(\sqrt{f \cdot d} + 120, 1800)$. For comparability, the definition of the dynamic timeframe within a range from 120 s until 1800 s is derived from the static experiments minimum and maximum times (2 and 30 min). Subsequently, the Stopping Tolerance $S$ can be determined based on $t_{train}$ and $n_m$: $S = \frac{1}{t_{train}} * \sqrt{\frac{1}{n_m}}$.

<footnote>$^5$ https://www.timeseriesclassification.com.</footnote>
<table><caption>**Table 2.** Summary of datasets used for evaluation as presented by Bagnall et al. [1].</caption><thead><tr><th>Dataset</th><th>Source</th><th>Train</th><th>Test</th><th>Dim</th><th>Length</th><th>Classes</th><th>Included</th><th>Comments</th></tr></thead><tbody><tr><td>AWR</td><td>coordinates</td><td>175</td><td>300</td><td>9</td><td>144</td><td>25</td><td>✓</td><td></td></tr><tr><td>AF</td><td>electrical biosignal</td><td>15</td><td>15</td><td>2</td><td>640</td><td>3</td><td>✓</td><td></td></tr><tr><td>BM</td><td>accelerometer</td><td>40</td><td>40</td><td>6</td><td>100</td><td>4</td><td>✓</td><td></td></tr><tr><td>CR</td><td>accelerometer</td><td>108</td><td>72</td><td>6</td><td>1197</td><td>12</td><td>✓</td><td></td></tr><tr><td>DDG</td><td>audio</td><td>50</td><td>50</td><td>1345</td><td>270</td><td>5</td><td>✗</td><td>deviating data format</td></tr><tr><td>EW</td><td>others</td><td>128</td><td>131</td><td>6</td><td>17984</td><td>5</td><td>✗</td><td>maximum time exceeded</td></tr><tr><td>EP</td><td>accelerometer</td><td>137</td><td>138</td><td>3</td><td>206</td><td>4</td><td>✓</td><td></td></tr><tr><td>EC</td><td>others</td><td>261</td><td>263</td><td>3</td><td>1751</td><td>4</td><td>✓</td><td></td></tr><tr><td>ER</td><td>others</td><td>30</td><td>270</td><td>4</td><td>65</td><td>6</td><td>✗</td><td>deviating data format</td></tr><tr><td>FD</td><td>electrical biosignal</td><td>5890</td><td>3524</td><td>144</td><td>62</td><td>2</td><td>✗</td><td>maximum time exceeded</td></tr><tr><td>FM</td><td>electrical biosignal</td><td>316</td><td>100</td><td>28</td><td>50</td><td>2</td><td>✓</td><td></td></tr><tr><td>HMD</td><td>electrical biosignal</td><td>160</td><td>74</td><td>10</td><td>400</td><td>4</td><td>✓</td><td></td></tr><tr><td>HW</td><td>accelerometer</td><td>150</td><td>850</td><td>3</td><td>152</td><td>26</td><td>✓</td><td></td></tr><tr><td>HB</td><td>audio</td><td>204</td><td>205</td><td>61</td><td>405</td><td>2</td><td>✓</td><td></td></tr><tr><td>LIB</td><td>coordinates</td><td>180</td><td>180</td><td>2</td><td>45</td><td>15</td><td>✓</td><td></td></tr><tr><td>LSST</td><td>others</td><td>2459</td><td>2466</td><td>6</td><td>36</td><td>14</td><td>✓</td><td></td></tr><tr><td>MI</td><td>electrical biosignal</td><td>278</td><td>100</td><td>64</td><td>3000</td><td>2</td><td>✗</td><td>maximum time exceeded</td></tr><tr><td>NATO</td><td>accelerometer</td><td>180</td><td>180</td><td>24</td><td>51</td><td>6</td><td>✓</td><td></td></tr><tr><td>PD</td><td>coordinates</td><td>7494</td><td>3498</td><td>2</td><td>8</td><td>10</td><td>✓</td><td></td></tr><tr><td>PEMS</td><td>others</td><td>267</td><td>173</td><td>963</td><td>144</td><td>7</td><td>✗</td><td>maximum time exceeded</td></tr><tr><td>PS</td><td>audio</td><td>3315</td><td>3353</td><td>11</td><td>217</td><td>39</td><td>✓</td><td></td></tr><tr><td>RS</td><td>accelerometer</td><td>151</td><td>152</td><td>6</td><td>30</td><td>4</td><td>✓</td><td></td></tr><tr><td>SRS1</td><td>electrical biosignal</td><td>268</td><td>293</td><td>6</td><td>896</td><td>2</td><td>✓</td><td></td></tr><tr><td>SRS2</td><td>electrical biosignal</td><td>200</td><td>180</td><td>7</td><td>1152</td><td>2</td><td>✓</td><td></td></tr><tr><td>SWJ</td><td>electrical biosignal</td><td>12</td><td>15</td><td>4</td><td>2500</td><td>3</td><td>✓</td><td></td></tr><tr><td>UW</td><td>accelerometer</td><td>120</td><td>320</td><td>3</td><td>315</td><td>8</td><td>✓</td><td></td></tr></tbody></table>

Experiments were run on consumer-grade hardware (e.g., Lenovo ThinkPad T14s laptop) without dedicated GPUs, in order to simulate real world working conditions and demonstrate the low barrier to entry of the *AutoTiM* service.

## 4.3 Results and Benchmark Comparison

A comparison of the achieved model performances and the benchmark results taken from [15] is presented in Table 3, whereby models were trained for all selected datasets from Table 2. The results show that *AutoTiM* delivers a strong performance for most of the tested datasets without using any kind of expert or domain knowledge. All in all, the results were better or on par with the respective benchmark results for 7 out of 20 datasets: *BM*, *EC*, *FM*, *LIB*, *LSST*, *SRS2*, and *SWJ*. For 7 datasets, the results are still very competitive (maximum deviation 3%): *AWR*, *CR*, *EP*, *HB*, *PD*, *RS*, and *SRS1* (4.2%). The results for 2 datasets are clearly poorer but still usable for classification: *NATO* and *PS*. Finally, the models of 4 datasets are significantly worse and do not appear useful for a production use-case: *AF*, *HMD*, *HW*, and *UW*.
<table><caption>**Table 3.** Classification accuracies (ACC) achieved with *AutoTiM* compared to the benchmark results presented in [15]. For the benchmark results, the average accuracy of 30 runs for the best-performing classification algorithm (Alg.) and the average accuracy of all trained algorithms per dataset are given. For *AutoTiM*, we provide the best accuracy and the average accuracy of multiple runs for each of the four experiments (see Subsect. 4.2).</caption><thead><tr><th rowspan="3"></th><th colspan="2">Benchmark results [15]</th><th colspan="8">AutoTiM results</th></tr><tr><th rowspan="2">Ø Best<br>Alg.</th><th rowspan="2">Ø All<br>Algs.</th><th colspan="2">2 min</th><th colspan="2">10 min</th><th colspan="2">30 min</th><th colspan="2">dynamic limit</th></tr><tr><th>Best<br>ACC</th><th>Ø<br>ACC</th><th>Best<br>ACC</th><th>Ø<br>ACC</th><th>Best<br>ACC</th><th>Ø<br>ACC</th><th>Best<br>ACC</th><th>Ø<br>ACC</th></tr></thead><tbody><tr><td>AWR</td><td>0.996</td><td>0.979</td><td>0.957</td><td>0.856</td><td>0.907</td><td>0.897</td><td>0.920</td><td>0.873</td><td>0.967</td><td>0.967</td></tr><tr><td>AF</td><td>0.740</td><td>0.305</td><td>0.333</td><td>0.260</td><td>0.333</td><td>0.293</td><td>0.333</td><td>0.240</td><td>0.267</td><td>0.227</td></tr><tr><td>BM</td><td>1.0</td><td>0.977</td><td>1.0</td><td>0.985</td><td>1.0</td><td>0.98</td><td>1.0</td><td>0.975</td><td>1.0</td><td>0.98</td></tr><tr><td>CR</td><td>1.0</td><td>0.986</td><td>0.986</td><td>0.962</td><td>0.986</td><td>0.975</td><td>0.986</td><td>0.978</td><td>0.986</td><td>0.947</td></tr><tr><td>EP</td><td>1.0</td><td>0.970</td><td>0.993</td><td>0.98</td><td>0.978</td><td>0.970</td><td>0.964</td><td>0.949</td><td>0.986</td><td>0.984</td></tr><tr><td>EC</td><td>0.824</td><td>0.490</td><td>0.981</td><td>0.701</td><td>0.981</td><td><strong>0.835</strong></td><td>0.981</td><td><strong>0.835</strong></td><td>0.441</td><td>0.416</td></tr><tr><td>FM</td><td>0.561</td><td>0.542</td><td>0.620</td><td>0.536</td><td>0.600</td><td><strong>0.566</strong></td><td>0.580</td><td>0.508</td><td>0.570</td><td>0.540</td></tr><tr><td>HMD</td><td>0.522</td><td>0.373</td><td>0.243</td><td>0.243</td><td>0.243</td><td>0.243</td><td>0.257</td><td>0.254</td><td>0.365</td><td>0.365</td></tr><tr><td>HW</td><td>0.657</td><td>0.467</td><td>0.118</td><td>0.090</td><td>0.124</td><td>0.104</td><td>0.126</td><td>0.099</td><td>0.111</td><td>0.098</td></tr><tr><td>HB</td><td>0.765</td><td>0.719</td><td>0.761</td><td>0.730</td><td>0.766</td><td>0.684</td><td>0.761</td><td>0.718</td><td>0.741</td><td>0.735</td></tr><tr><td>LIB</td><td>0.941</td><td>0.868</td><td>0.983</td><td><strong>0.968</strong></td><td>0.972</td><td>0.863</td><td>0.922</td><td>0.813</td><td>0.856</td><td>0.839</td></tr><tr><td>LSST</td><td>0.637</td><td>0.532</td><td>0.982</td><td><strong>0.980</strong></td><td>0.985</td><td><strong>0.982</strong></td><td>0.981</td><td><strong>0.740</strong></td><td>0.695</td><td><strong>0.690</strong></td></tr><tr><td>NATO</td><td>0.971</td><td>0.852</td><td>0.739</td><td>0.705</td><td>0.739</td><td>0.666</td><td>0.739</td><td>0.713</td><td>0.739</td><td>0.711</td></tr><tr><td>PD</td><td>0.997</td><td>0.974</td><td>0.975</td><td>0.960</td><td>0.973</td><td>0.925</td><td>0.975</td><td>0.944</td><td>0.975</td><td>0.974</td></tr><tr><td>PS</td><td>0.329</td><td>0.241</td><td>0.245</td><td>0.234</td><td>0.258</td><td>0.250</td><td>0.265</td><td>0.238</td><td>0.259</td><td>0.257</td></tr><tr><td>RS</td><td>0.930</td><td>0.887</td><td>0.914</td><td>0.863</td><td>0.908</td><td>0.895</td><td>0.875</td><td>0.836</td><td>0.868</td><td>0.868</td></tr><tr><td>SRS1</td><td>0.957</td><td>0.832</td><td>0.901</td><td>0.881</td><td>0.915</td><td>0.888</td><td>0.894</td><td>0.870</td><td>0.908</td><td>0.887</td></tr><tr><td>SRS2</td><td>0.536</td><td>0.508</td><td>0.456</td><td>0.456</td><td>0.456</td><td>0.456</td><td>0.456</td><td>0.456</td><td>0.578</td><td><strong>0.537</strong></td></tr><tr><td>SWJ</td><td>0.464</td><td>0.377</td><td>0.733</td><td><strong>0.513</strong></td><td>0.467</td><td>0.347</td><td>0.533</td><td>0.427</td><td>0.467</td><td>0.400</td></tr><tr><td>UW</td><td>0.949</td><td>0.896</td><td>0.219</td><td>0.192</td><td>0.216</td><td>0.191</td><td>0.209</td><td>0.194</td><td>0.681</td><td>0.679</td></tr></tbody></table>

## 4.4 Discussion

Overall, the results of *AutoTiM* proved to be better or at least highly competitive for 14 out of 20 datasets. Concerning the poor performance of *AutoTiM* for *AF*, *HMD*, *HW*, and *UW*: except for *UW* with an accuracy of more than 94%, the average benchmark performance for the other 3 datasets is also significantly low. This indicates that the underlying data is complex and not easy to classify without domain-specific tailoring, which is out of scope for *AutoTiM*. The data sources for both *AF* and *HMD* are electrical biosignals, while *HW* as well as *UW* encompass accelerometer data. However, both groups of data are also covered by other datasets for which *AutoTiM* trained highly performant models. This implies that trained models are not dependent on underlying categories or types of data.

A qualitative evaluation as well as a more critical look at the service is possible if the requirements from Sect. 3.1 are taken into consideration. Regarding
points 1 to 4, *AutoTiM* meets all requirements as it is an end-to-end service for automated training, evaluation, versioning, and deployment of machine learning models for time-series-based data, available both locally and in cloud-based environments. Moreover, it is manageable via three standardized and parameterized REST interfaces; no domain expertise is required to build highly capable prediction models within minutes. In contrast and as discussed above, the requirements of points 5 and 6 from Sect. 3.1 regarding *Adaptability* and *Model Quality* are not fully met yet.

# 5 Summary and Future Work

Within this paper, we presented *AutoTiM*, a holistic end-to-end automated machine learning service that includes all necessary technical components and logical steps for training, evaluation, and provisioning of supervised classification models for time series data. In addition, the service is available as an environment-independent open-source application, which makes it particularly interesting for small and medium-sized enterprises, as it enables a low-cost entry to the development and operation of machine learning applications. As such, it can be used in local environments for rapid prototyping as well as in productive, cloud-based machine learning applications, e.g., as an automated retraining component in complex *MLOps* architectures. Section 4 shows that the performance of *AutoTiM* is mostly competitive and sometimes even outperforming other approaches, which conduct steps like feature engineering, training, and tuning in a manual fashion. Not only is this time-consuming, but also requires lots of domain expertise. In contrast to that, *AutoTiM* only needs a few minutes for training and no domain knowledge.

The release of *AutoTiM* is a first step: there are still open issues that will be addressed by the authors in the upcoming research project *DeKIOps*<sup>6</sup>. Contributions from the open-source community are welcome and encouraged. Currently, the service focuses on time series data, so preprocessing raw data may be challenging. Thereby, the formalization of existing and the standardization of new preprocessing modules would facilitate compatibility with heterogeneous data sources. Additionally, the number of supported algorithms must be extended to broaden the range of applications of *AutoTiM* in productive scenarios. Another area of improvement lies within the classification performance itself. The results in Sect. 4 illustrate that competitive results can be achieved without domain knowledge, but also showcase use-cases where meaningful data patterns could not be detected automatically. Therefore, it is reasonable to explore ways to facilitate (but not require) injection of existing knowledge into an automated service such as *AutoTiM*. Bringing these unresolved issues into the open-source community and thus promoting the creation of a mature generic service available to everyone with no restrictions is one of the authors’ main concerns within the upcoming *DeKIOps* research project.

<footnote n="6"><sup>6</sup> *DeKIOps* - Democratization of AI With Explainable and Easily Accessible Machine Learning Operations.</footnote>
References

1. Bagnall, A., et al.: The UEA multivariate time series classification archive. arXiv preprint (2018). https://doi.org/10.48550/arXiv.1811.00075

2. Carmona, C.U., Aubet, F.X., Flunkert, V., Gasthaus, J.: Neural contextual anomaly detection for time series. In: IJCAI-2022, pp. 2843–2851 (2022). https://doi.org/10.24963/ijcai.2022/394

3. Carvalho, T.P., Soares, F.A., Vita, R., Francisco, R.P., Basto, J.P., Alcalá, S.G.: A systematic literature review of machine learning methods applied to predictive maintenance. CAIE **137**, 106024 (2019). https://doi.org/10.1016/j.cie.2019.106024

4. Coelho, D., Costa, D., Rocha, E.M., Almeida, D., Santos, J.P.: Predictive maintenance on sensorized stamping presses by time series segmentation, anomaly detection, and classification algorithms. Procedia Comput. Sci. **200**, 1184–1193 (2022). https://doi.org/10.1016/j.procs.2022.01.318

5. Davari, N., Veloso, B., de Assis Costa, G., Pereira, P.M., Ribeiro, R.P., Gama, J.: A survey on data-driven predictive maintenance for the railway industry. Sensors **21**(17), 5739 (2021). https://doi.org/10.3390/s21175739

6. Esling, P., Agon, C.: Time-series data mining. ACM Comput. Surv. **45**(1), 1–34 (2012). https://doi.org/10.1145/2379776.2379788

7. Ferreira, L., Pilastri, A., Martins, C.M., Pires, P.M., Cortez, P.: A comparison of AutoML tools for machine learning, deep learning and XGBoost, pp. 1–8. IEEE (2021). https://doi.org/10.1109/IJCNN52387.2021.9534091

8. Giebler, C., Gröger, C., Hoos, E., Schwarz, H., Mitschang, B.: Leveraging the data lake: current state and challenges. In: Ordonez, C., Song, I.-Y., Anderst-Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DaWaK 2019. LNCS, vol. 11708, pp. 179–188. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27520-4_13

9. Jardine, A.K., Lin, D., Banjevic, D.: A review on machinery diagnostics and prognostics implementing condition-based maintenance. MSSP **20**(7), 1483–1510 (2006). https://doi.org/10.1016/j.ymssp.2005.09.012

10. Karmaker (“Santu”), S.K., Hassan, M.M., Smith, M.J., Xu, L., Zhai, C., Veeramachaneni, K.: AutoML to date and beyond: challenges and opportunities. ACM Comput. Surv. **54**(8), 1–36 (2021). https://doi.org/10.1145/3470918

11. LeDell, E., Poirier, S.: H2O AutoML: scalable automatic machine learning. In: 7th ICML Workshop on Automated Machine Learning (AutoML) (2020)

12. Patel, D., Ganapavarapu, G., Jayaraman, S., Lin, S., Bhamidipaty, A., Kalagnanam, J.: AnomalyKiTS: anomaly detection toolkit for time series, vol. 36, pp. 13209–13211 (2022)

13. Polge, J., Robert, J., Le Traon, Y.: A case driven study of the use of time series classification for flexibility in industry 4.0. Sensors **20**(24), 7273 (2020). https://doi.org/10.3390/s20247273

14. Roblek, V., Meško, M., Krapež, A.: A complex view of industry 4.0. SAGE Open **6**(2), 2158244016653987 (2016). https://doi.org/10.1177/2158244016653987

15. Ruiz, A.P., Flynn, M., Large, J., Middlehurst, M., Bagnall, A.: The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Disc. **35**(2), 401–449 (2020). https://doi.org/10.1007/s10618-020-00727-3

16. Shaukat, K., et al.: A review of time-series anomaly detection techniques: a step to future perspectives. In: Arai, K. (ed.) FICC 2021. AISC, vol. 1363, pp. 865–877. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73100-7_60
17. Smith, M.J., Sala, C., Kanter, J.M., Veeramachaneni, K.: The machine learning bazaar: harnessing the ML ecosystem for effective system development, pp. 785–800 (2020). https://doi.org/10.1145/3318464.3386146

18. Zaharia, M., et al.: Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull. **41**(4), 39–45 (2018)

19. Zöller, M.A., Huber, M.F.: Benchmark and survey of automated machine learning frameworks. JAIR **70**, 409–472 (2021). https://doi.org/10.1613/jair.1.11854