# CaTabRa: Efficient Analysis and Predictive Modeling of Tabular Data

Alexander Maletzky, Sophie Kaltenleithner, Philipp Moser, Michael Giretzlehner

► To cite this version:

Alexander Maletzky, Sophie Kaltenleithner, Philipp Moser, Michael Giretzlehner. CaTabRa: Efficient Analysis and Predictive Modeling of Tabular Data. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.57-68, 10.1007/978-3-031-34107-6_5. hal-05498072

HAL Id: hal-05498072

https://inria.hal.science/hal-05498072v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# CaTabRa: Efficient Analysis and Predictive Modeling of Tabular Data

Alexander Maletzky(✉)D, Sophie KaltenleithnerD, Philipp MoserD, and Michael GiretzlehnerD

Research Department Medical Informatics, RISC Software GmbH, Softwarepark 32a,
4232 Hagenberg, Austria
{alexander.maletzky,sophie.kaltenleithner,philipp.moser,
michael.giretzlehner}@risc-software.at

**Abstract.** We present CaTabRa, a novel open-source Python package for the efficient and largely automated analysis of tabular data. It combines a variety of established frameworks and libraries for data processing, automated machine learning, explainable AI and out-of-distribution detection into one coherent system. Thanks to its simple user interface, CaTabRa can be used by practitioners who want to quickly gain insights into their data and the potential of predictive modeling, but it also provides added value for data-science experts through its function library. We demonstrate CaTabRa's usefulness in two example applications.

**Keywords:** Machine learning · Tabular data · AutoML · OOD detection · Explainable AI

## 1 Introduction

Tabular data are ubiquitous in many domains, including the health sector, business intelligence, industrial applications, empirical research, and many more. Effectively and efficiently utilizing these data, e.g. for predictive and prescriptive modeling, for a long time required profound expertise in data science. While this remains true to a certain extent, in recent years there has been a trend towards automating ever more aspects of data science, both to generate more and better output in shorter time, but also to make it accessible to non-experts. Automation affects machine learning (AutoML [4, 11]), feature engineering [7, 17] and report generation, implemented in frameworks, systems and tools of different kinds (Sect. 1.1). Yet, we found none of them offer the right degree of automation for being able to efficiently solve problems of varying complexity.

Therefore, our goal was to develop a user-friendly data analysis system that supports simple and complex tasks alike, and as such provides added value for both non-experts (or *domain experts*) and experienced data science professionals. Simple tasks should be handled in an entirely automatic way, to quickly gain insights into data and the potential of predictive modeling; complex tasks should be solvable by combining powerful data wrangling functions to clean,
efficient programs. The resulting *CaTabRa* system builds upon established tools and frameworks for data processing, automated machine learning (AutoML), explainable AI and out-of-distribution detection (OOD-detection). It integrates them into a coherent whole, adding glue code [30] where necessary and implementing useful functionality missing in the constituent frameworks as readily available Python functions.

CaTabRa's design is inspired by one key principle: *Automate what can reasonably be automated, don't try to automate everything*. Unlike many related systems, CaTabRa neither claims nor aims to automate all aspects of the typical data analysis workflow. With increasing data complexity, according to our own experience, some programming and data-science skills will always be necessary to obtain high-quality prediction models from raw data [19]. Above all this concerns the definition and extraction of meaningful target labels for supervised learning. Even if all feature engineering can be automated, as claimed in [17], label extraction generally cannot. Especially in the medical domain, label- and sample definitions can become fairly complicated (see [14] for an example) and do not fit into predefined 'templates' offered by related systems like [2].

CaTabRa is both a ready-to-use command-line tool and a function library. The command-line interface automates the parts of the data analysis workflow that can reasonably be automated, and the function library exposes the functionality of the command-line interface through a simple API and furthermore provides building blocks for the steps that cannot be automated easily.

Summarizing, our main contributions in this paper are:

- A novel, open-source tabular data analysis system and library (Sect. 2), available at https://github.com/risc-mi/catabra.

- A largely automated workflow (Sect. 2.1) encompassing not only model training (Sect. 2.2), but also model evaluation and explanation (Sect. 2.4), OOD-detection (Sect. 2.5), and the creation of publication-ready descriptive statistics and visualizations (Sect. 2.3).

- Functions for efficient data preparation and feature extraction, especially from longitudinal data (Sect. 2.6).

- Two specific, real-world use-cases for demonstrating the practical applicability and usefulness of the system (Sect. 3).

## 1.1 Related Work

Frameworks similar to CaTabRa have been proposed in the past for different goals and end users. A brief overview of these solutions including their main differences to CaTabRa is given below; a more thorough comparison of CaTabRa to related work is not in the scope of this paper.

*PyCaret* [1] is perhaps the system most similar to CaTabRa. The Python library is essentially a wrapper around well-known machine learning libraries and frameworks, such as scikit-learn, optuna and spaCy. It aims to reduce the technological knowledge necessary to perform moderately difficult data analysis tasks (supervised and unsupervised). In contrast to CaTabRa's default auto-sklearn AutoML backend, data preprocessing is not considered during optimization and more manual
effort is required for executing a complete pipeline including data analysis, model
training and evaluation. MLJAR [26] is another similar solution focusing on super-
vised machine learning tasks. It aims at reducing development time for data scientists
by offering multiple training modes depending on whether the goal is explainabil-
ity, a production-ready pipeline or a highly-tuned model. It includes preprocessing,
ensemble-construction, hyperparameter tuning, evaluation and explanation. How-
ever, including custom building blocks (e.g., new model classes) is not straightfor-
ward. AMLBID [12] is a Python framework focusing on facilitating model-training
and analysis for non-data-scientists. It consists of two main components: A recom-
mender that proposes a training pipeline based on meta-learning and an explainer
that starts a dashboard providing data level explanations (correlations, missing val-
ues), model level explanations (feature importance, decision paths) and performance
evaluations. As algorithm selection is done by meta learning, where hyperparame-
ters are guessed from experiments with other datasets, the user is limited to the eight
algorithms built into the system. Further, preprocessing is not included in the opti-
mization pipeline.

Besides these generic solutions, some more specialized ones have been pro-
posed in the medical field: *Cardea* [2] is a general end-to-end framework special-
ized on electronic health records. It takes data from a relational database and
applies automatic feature engineering. Prediction models are generated using *ML
Bazaar* [32] and hyperopt [4] for a fixed set of problem definitions, which limits
its general applicability. Performance reports are not as thorough as in CaTabRa
and an explanation component is not included by default. *Clairvoyance* [15] is a
generic pipeline for processing time-series medical data and training prediction
models including model building and hyperparameter optimization. The system
lacks flexibility due to implicit assumptions on the format of the input data and
limited options for defining target labels. *AutoDC* [3] is a framework for disease
classification based on gene expression data using AutoML. While it comprises
many interesting functionalities (e.g., optimizations of preprocessing steps and
model training, model explanations), its focus on a specific application makes it
unusable for other tasks without extensive code refactoring.

Cloud-based solutions like Google's Vertex AI [13] and Azure Machine Learn-
ing [21] are platform-dependent, not free-to-use or open-source and require data
to be uploaded to external servers, which could be problematic in terms of data
protection. We consider these solutions to be aimed at a different target group.

In summary, while solutions exist that resemble CaTabRa's functionality in
some ways, they differ in the degree of automation, flexibility, extensibility and
accessibility. Also, none of them considers the applicability of prediction models
to new data, as CaTabRa does through its OOD detection component (Sect. 2.5).

2 The CaTabRa System

2.1 General Workflow

CaTabRa covers all aspects of tabular data analysis, ranging from generating
descriptive statistics over training prediction models, evaluating and explaining
<figure><img src="image_3.png" /><figcaption>Fig. 1. CaTabRa workflow.</figcaption></figure>

them on test data to applying them to new data for making predictions. Although data preparation (transforming raw, unstructured/semi-structured data into tabular form) is not part of CaTabRa's largely automated core functionality by design, the system also provides some ready-to-use data preparation tools through its function library (Sect. 2.6).

The general CaTabRa workflow is summarized in Fig. 1. It essentially consists of four main steps: *analyze*, *evaluate*, *explain* and *apply*. Each of these steps can be easily invoked both through CaTabRa's command-line interface and through corresponding library functions, and requires relatively little user input.

*Analyze* takes a single data table, with rows corresponding to samples and columns corresponding to attributes, as input and thoroughly analyzes it. This includes generating descriptive statistics for each attribute, training prediction models if one or more target attributes are specified (Sect. 2.2), setting up model explainers (Sect. 2.4) and creating OOD detection models (Sect. 2.5). All generated artifacts are stored in a local directory that serves as input for the other steps.

*Evaluate* applies prediction models trained in *analyze* to test data and creates thorough performance reports, both in tabular form and as static and/or interactive plots (Sect. 2.3). It automatically ensures that the given test data has the same format as the data originally analyzed, and in addition applies the OOD detector to identify potential issues in the data distribution.

*Explain* explains the trained prediction model. The required input and generated output largely depend on the used backend (Sect. 2.4).

*Apply* applies the trained model to (unlabeled) data, generating predictions and OOD-scores. It optionally explains how the model arrived at its predictions.

## 2.2 Model Training and Hyperparameter Tuning: AutoML

CaTabRa leverages state-of-the-art automated machine learning (AutoML) systems to quickly produce top-performing prediction models for four widespread prediction tasks (binary/multi-class/multilabel classification, regression) without cumbersome manual pipeline configuration and hyperparameter tuning. The default AutoML backend is auto-sklearn [10, 11], which constructs and optimizes whole pipelines, including data- and feature preprocessing (imputation, encoding, feature selection, etc.) and the final ML algorithm. Other backends can be
incorporated easily if auto-sklearn's built-in capabilities turn out insufficient for a given use-case.

It is of course also possible to extend auto-sklearn itself, for instance by adding preprocessing steps or machine learning algorithms. CaTabRa by default adds extreme gradient boosting (XGBoost) [6] to the arsenal of readily available machine learning algorithms, and provides convenience features that facilitate interacting with auto-sklearn. For example, it implements additional resampling strategies for internal validation during hyperparameter optimization, in particular, train-test splits where samples can be assigned to groups and all samples of a group are put into the same data partition ('grouped splitting'). Another convenience feature is the ability to abort hyperparameter optimization at any time through simple keyboard interrupts, and still combine the models trained so far to a final ensemble.

Classification models are often ill-calibrated even if they perform well in terms of area under the receiver operating characteristic curve and related metrics [23]. Therefore, classifiers created with CaTabRa can be calibrated easily, either immediately during model training by specifying the subset of the data to calibrate on, or through a stand-alone command/function.

## 2.3 Performance Reports

CaTabRa automatically generates extensive performance reports when evaluating trained prediction models on test data, consisting of both tables with numerical data and static and/or interactive plots. Numerical results include all suitable performance metrics available in scikit-learn [24], with micro-, macro- and weighted averaging for prediction tasks with multiple targets. Classification metrics that depend on a specific decision threshold rather than a continuous probability (like precision and recall) are by default evaluated for multiple thresholds to allow users select the optimal threshold based on their own preference. In addition, several widely used strategies for selecting decision thresholds based on objective criteria are implemented as well, for instance, minimizing the difference between sensitivity and specificity, the closest-to-(0, 1) criterion [25], or maximizing any given metric. Bootstrapping [9] is an integral part of CaTabRa's performance evaluation tools as well, which can be activated on demand and returns both aggregate (mean, standard deviation, etc.) and detailed information about the corresponding random subsamples of the test data.

Model performance is visualized in publication-ready static plots and interactive plots generated with plotly [27], without any user interaction required. The kind of data visualized depends on the prediction task, but includes all common plot types like scatter plots of ground truth vs. predictions in regression settings, ROC- and precision-recall (PR) curves in binary classification, and fusion matrices in binary- and multiclass classification. Examples can be found in Sect. 3.1.
## 2.4 Model Explanations

CaTabRa uses SHAP [18] as its default model explanation framework. SHAP explains models by calculating sample-wise feature importance values and works particularly well on tabular data. It was thus the natural choice for our system, but other explanation backends can be added, too.

Since SHAP is able to explain any (black-box) machine learning model it can be readily applied to ensembles and pipelines of preprocessing steps and final estimators, two model-types frequently returned by AutoML systems. This approach has two downsides, though: it is relatively slow and only gives approximate SHAP values. Fast and exact methods exist for tree-based models, like random forests, but cannot be applied to pipelines that include preprocessing steps. The overcome this limitation CaTabRa implements rules for back-propagating feature importance through some of the most common preprocessing transformations, like rescaling, one-hot encoding, or feature selection. This means that dedicated explanation techniques can be applied despite the presence of (some types of) data preprocessing.

Similar to the performance reports described in Sect. 2.3, explaining a model automatically creates visually appealing plots of the obtained feature importance.

## 2.5 Out-of-Distribution Detection

Machine learning models generally assume that test data follows the same distribution as the data they were initially trained on. If this assumption is violated, for instance due to semantic- or covariate shifts, the models' predictions can become unreliable. CaTabRa therefore integrates a set of OOD detectors to identify samples that are likely OOD. They are trained during initial data analysis and automatically applied when dealing with new (unseen) data.

CaTabRa offers five methods for OOD detection: *Bins Detector*: A simple detector that divides each feature into equally sized bins. A new value that falls within a bin without any instances in the training data is considered OOD.
* *Autoencoder*: A neural network that consists of an encoder, which maps features to a lower dimensional space, and a decoder, which aims to reconstruct the original values. This assumes that OOD samples have a higher reconstruction error than in-distribution (ID) samples [34].
* *Soft Brownian Offset Detector* [22]: A detector based on a method to generate synthetic OOD samples from the training data. Once they have been generated, a classifier is trained to distinguish the OOD and ID samples.
* *Kolmogorov-Smirnov-Test* (KS-Test) [31]: A statistical test assessing whether two samples are likely to stem from the same distribution.
* *PyOD Detector*: A wrapper for algorithms included in the PyOD [33] framework. This framework contains a collection of classes for outlier and anomaly detection which can double as OOD detectors.
## 2.6 Utilities for Working with Longitudinal Data

CaTabRa offers a couple of useful tools for transforming raw data into the tabular form required by its core workflow. One of these tools allows to efficiently work with large amounts of irregularly-sampled longitudinal data, as can be often found in electronic health records, for instance. Each entry in such data corresponds to an observation of a certain attribute for a certain entity (e.g., subject) at a given time. Converting longitudinal data into the ‘samples × features’ format usually amounts to aggregating over the temporal dimension, e.g., by computing the first, last, average, etc. observed value of each entity and attribute over time. Many standard machine learning libraries (like Pandas [20] and tsfresh [7]) implement related functions, but they all lack support for *efficiently aggregating* values over a given set of (non-uniform, overlapping) time windows individually: assuming the observations are stored in a table in long table format and the windows are contained in a table as well, with columns for entity-identifiers, start- and end times, aggregate observed values per time window and store the results in a table with one row per window and one column per attribute. The solution implemented in CaTabRa consists of a function with a simple interface, that takes the tables of observations and windows, and the requested aggregations, as input, and returns the output table as described above. It is implemented highly efficiently and even supports parallel and distributed execution through Dask DataFrames [8]. An example use-case is sketched in Sect. 3.2.

# 3 Use-Cases

## 3.1 Use-Case 1: COVID-19 Prediction

Roland et al. have recently evaluated machine learning models for the prediction of COVID-19 diagnosis based on routinely acquired blood tests [29]. By comparing evaluation metrics obtained with different train-test splits, they showed (a) that their data exhibited domain shifts over time (e.g., changes in disease prevalence, testing procedures and virus mutations) and (b) that these had diminishing effects on the predictive performance and reliability of COVID-19 diagnosis predictions. The dataset contained 127,115 samples (1.24% positives) and included 100 features ranging from age and sex to various blood test parameters (e.g., leucocyte count).

Since CaTabRa was not used in the original publication, we showcase in this use-case how CaTabRa allows to tackle this supervised classification problem, covering all steps from initial data inspection to final model evaluation. We opted to reproduce only the temporal split scenario from the original publication, but the minimal coding effort required by CaTabRa would allow to easily extend our analysis to a variety of splits and classifiers. Using CaTabRa’s *analyze* functionality we first generated descriptive statistics about the dataset which also encompasses inferential tests for significant differences between the negative- and positive classes. For testing whether the expected domain shifts can be picked
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Performance figures for the Use-Case 1 (COVID-19 prediction) using the XGBoost classifier on the test set. CaTabRa produces a variety of publication-ready figures including ROC-AUC curves (the 95% confidence interval is shown as shaded area around the curve), confusion matrices and threshold-dependency plots for various metrics.</figcaption></figure>

up by the framework, the KS-test was used for OOD detection. This detector was chosen as it checks for an overall change of distribution in features in contrast to a per-sample approach. The model training including hyperparameter tuning described in Sect. 2.2 was performed with two classifier approaches: (a) an XGBoost trained for 15 min and (b) an ensemble of 25 classifiers trained for 120 min. The first was chosen as it outperformed other model classes in the original publication, while the ensemble should highlight CaTabRa's diversity in classifiers. After model training, CaTabRa's *evaluate* functionality was applied to the test set using 1000 repetitions of bootstrapping, thereby generating a comprehensive performance report (Sect. 2.3) comprised of various plots (Fig. 2) and performance metrics. Our results for the area under the ROC curve (ROC-AUC) were 0.8797 ± 0.0054 (XGBoost) and 0.8935 ± 0.0052 (ensemble). For the PR-AUC we obtained 0.6530 ± 0.0143 (XGBoost) and 0.6843 ± 0.0148 (ensemble). Our results compared favourably with the best results in [29], which were 0.8142 ± 0.0000 (ROC-AUC) and 0.7077 ± 0.0000 (PR-AUC). Further, the OOD detector created during the *analyze* phase was automatically applied during *evaluate*. 81 of 95 continuous features were flagged as stemming from a different distribution. In contrast, only 18 features were considered OOD when applying the same detector to a validation set sampled closer in time to the training set. This illustrates that CaTabRa successfully detected the domain shift reported in [29].

In summary, CaTabRa allowed us to easily implement machine learning models to predict COVID-19 diagnoses. Data loading, model training and evaluation were coded in about 10 lines of Python code in less than 30 min of work.

## 3.2 Use-Case 2: ICU Mortality Prediction

MIMIC-III is a large-scale public database with electronic health records of an intensive care unit (ICU) [16]. It has been used in a wide variety of data analysis projects, for instance ICU mortality prediction [5]. In order to objectively compare results produced by our framework to those presented in [5], we prepared
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Training and explanation plots for Use-Case 2 (ICU mortality prediction) Left: Train and validation metrics over time Right: Local feature explanations for one model on the test set.</figcaption></figure>

the raw data following the exact same preprocessing steps. In summary, this comprises (1) selecting a proper data subset based on patient age and length of ICU stay, (2) extracting medical measurements (laboratory tests, vital signs) in a certain time frame, and (3) data harmonization, validation and imputation. The resulting data set consisted of six static features and 16 dynamic features with one value per hour in the first 48 h of each ICU stay.

For extracting the measurements in (2), the longitudinal data preparation utilities from Sect. 2.6 could be employed. We then again used CaTabRa's *analyze* functionality to train prediction models. In this scenario, an ensemble of 5 classifiers was trained. Different training times were configured with 10, 30, 60 and 120 min. During training, CaTabRa logs model performance and creates corresponding plots (Fig. 3) for inspecting the training process. This revealed that the number of trained ensembles increased from 0 to 7 with longer training, explaining the test-set ROC-AUC scores obtained when running *evaluate* with 1000 bootstrapping repetitions: 0.5 ± 0.0000, 0.8445 ± 0.0090, 0.8842 ± 0.0079 and 0.8866 ± 0.0084. A longer training time allowed more models to be trained, which lead to better results. However, a good performance could be achieved in less than an hour. The quality of our best model is comparable to the one of [5], with a ROC-AUC of 0.8735 ± 0.0025.

A central aspect of [5] is explaining the prediction models in terms of feature importance, something supported by CaTabRa out of the box (Sect. 2.4). At the end of the initial *analyze* phase, a SHAP explainer was automatically fitted on the final ensemble. In the *explain* phase this explainer is loaded and used for explaining the ensemble on a given data set. For each model in the ensemble feature importance scores are calculated and visualized in the form of a so-called 'beeswarm' plot. Figure 3 shows one for a random forest classifier part of the ensemble trained for 120 min. The ten most important features as well as the total importance of remaining features are depicted. Each dot corresponds to a sample with its color indicating the feature value. The three components of the Glasgow Coma Scale all rank highly compared to other features. This score
assesses a patient's level of consciousness and is already employed in research and ICU scoring systems [28], suggesting that the model relies on clinically meaningful features.

## 4 Conclusion

We presented CaTabRa, a novel software tool and Python library for analyzing tabular data and training high-quality prediction models with little effort. Thanks to its key design principle it can be readily used by domain experts for tackling simple machine learning tasks and at the same time provides added value for experienced Python developers in large-scale data science projects. We successfully used it in our own projects already, for instance in predicting organ failure after blood transfusions, determining whether patients discharged from an intensive care unit will be readmitted within a short time, and predicting functional outcomes after surgical clipping of brain aneurysms.<fnref n="1" /> Although these projects and the two use cases in Sect. 3 are concerned with medical data analysis, CaTabRa itself can be applied to tabular data from any domain. Examples can be found in the GitHub repository (https://github.com/risc-mi/catabra).

CaTabRa is ready to be used, but there are a couple of potential limitations that might have to be addressed in the future. For instance, integrating feature extraction (e.g., from longitudinal data) into the existing AutoML component is currently not easily possible, owing to the limited pipeline configuration capabilities of the default auto-sklearn backend. Second, big data and distributed computing are currently only fully supported by the data preparation tools described in Sect. 2.6, but corresponding functionality is yet to be integrated into the core workflow. And third, a graphical user interface could even further lower the entrance barrier for users unacquainted with Python and the command-line.

**Acknowledgements.** This project is financed by research subsidies granted by the government of Upper Austria. RISC Software GmbH is Member of UAR (Upper Austrian Research) Innovation Network.

We thank the anonymous reviewers for their valuable comments.

## References

1. Ali, M., et al.: PyCaret. https://pycaret.org/. Accessed 30 Mar 2023

2. Alnegheimish, S., et al.: Cardea: an open automated machine learning framework for electronic health records. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 536–545 (2020). https://doi.org/10.1109/DSAA49011.2020.00068

3. Bai, Y., Li, Y., Shen, Y., Yang, M., Zhang, W., Cui, B.: AutoDC: an automatic machine learning framework for disease classification. Bioinformatics 38(13), 3415–3421 (2022). https://doi.org/10.1093/bioinformatics/btac334

<footnote n="1">¹ Scientific publications describing these projects are currently in preparation.</footnote>
4. Bergstra, J., Yamins, D., Cox, D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, Atlanta, Georgia, USA, vol. 28, pp. 115–123. PMLR (2013). https://proceedings.mlr.press/v28/bergstra13.html

5. Caicedo-Torres, W., Gutierrez, J.: ISeeU: visually interpretable deep learning for mortality prediction inside the ICU. J. Biomed. Inform. **98**, 103269 (2019). https://doi.org/10.1016/j.jbi.2019.103269

6. Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2016, pp. 785–794. ACM, New York (2016). https://doi.org/10.1145/2939672.2939785

7. Christ, M., Braun, N., Neuffer, J., Kempa-Liehr, A.W.: Time series feature extraction on basis of scalable hypothesis tests (tsfresh - a python package). Neurocomputing **307**, 72–77 (2018). https://doi.org/10.1016/j.neucom.2018.03.067

8. Dask Development Team: Dask: Library for dynamic task scheduling (2016). https://dask.org. Accessed 30 Mar 2023

9. Efron, B.: Bootstrap methods: another look at the jackknife. Ann. Stat. **7**, 1–26 (1979). https://doi.org/10.1214/aos/1176344552

10. Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., Hutter, F.: Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. Technical report. arXiv:2007.04074 (2021)

11. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Advances in Neural Information Processing Systems, vol. 28. Curran Associates Inc. (2015). https://papers.neurips.cc/paper/2015/hash/11d0e6287202fced83f79975ec59a3a6-Abstract.html

12. Garouani, M., Ahmad, A., Bouneffa, M., Hamlich, M.: AMLBID: an auto-explained automated machine learning tool for big industrial data. SoftwareX **17**, 100919 (2022). https://doi.org/10.1016/j.softx.2021.100919

13. Google: Vertex AI. https://cloud.google.com/vertex-ai?hl=en. Accessed 30 Mar 2023

14. Hatib, F., et al.: Machine-learning algorithm to predict hypotension based on high-fidelity arterial pressure waveform analysis. Anesthesiology **129**(4), 663–674 (2018). https://doi.org/10.1097/ALN.00000000000002300

15. Jarrett, D., Bica, I., Ercole, A., Yoon, J., Qian, Z., van der Schaar, M.: Clairvoyance: a pipeline toolkit for medical time series. In: Proceedings of ICLR 2021, p. 32 (2021)

16. Johnson, A.E.W., Stone, D.J., Celi, L.A., Pollard, T.J.: The MIMIC code repository: enabling reproducibility in critical care research. J. Am. Med. Inform. Assoc. **25**(1), 32–39 (2018). https://doi.org/10.1093/jamia/ocx084

17. Kanter, J.M., Veeramachaneni, K.: Deep feature synthesis: towards automating data science endeavors. In: 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 1–10 (2015). https://doi.org/10.1109/DSAA.2015.7344858

18. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 4765–4774. Curran Associates Inc. (2017). https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
19. Maletzky, A., et al.: Lifting hospital electronic health record data treasures: challenges and opportunities. JMIR Med. Inform. 10(10), e38557 (2022). https://doi.org/10.2196/38557

20. McKinney, W.: Data structures for statistical computing in python. In: van der Walt, S., Millman, J. (eds.) Proceedings of the 9th Python in Science Conference, pp. 56–61 (2010). https://doi.org/10.25080/Majora-92bf1922-00a

21. Microsoft: Azure Machine Learning - ML as a Service. https://azure.microsoft.com/en-us/products/machine-learning. Accessed 30 Mar 2023

22. Müller, F., Botache, D., Huseljic, D., Heidecker, F., Bieshaar, M., Sick, B.: Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders. arXiv:2105.02965 (2021)

23. Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning, ICML 2005, pp. 625–632. Association for Computing Machinery, New York (2005). https://doi.org/10.1145/1102351.1102430

24. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

25. Perkins, N.J., Schisterman, E.F.: The inconsistency of ‘optimal’ cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am. J. Epidemiol. 163, 670–675 (2006). https://doi.org/10.1093/aje/kwj063

26. Płońska, A., Płoński, P.: Mljar: state-of-the-art automated machine learning framework for tabular data. version 0.10.3 (2021). https://github.com/mljar/mljar-supervised. Accessed 30 Mar 2023

27. Plotly Technologies Inc.: Collaborative data science (2015). https://plotly.com. Accessed 30 Mar 2023

28. Reith, F.C.M., Van den Brande, R., Synnot, A., Gruen, R., Maas, A.I.R.: The reliability of the Glasgow Coma Scale: a systematic review. Intensive Care Med. 42(1), 3–15 (2015). https://doi.org/10.1007/s00134-015-4124-3

29. Roland, T., et al.: Domain shifts in machine learning based Covid-19 diagnosis from blood tests. J. Med. Syst. 46(5), 1–12 (2022). https://doi.org/10.1007/s10916-022-01807-1

30. Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS 2015, pp. 2503–2511. MIT Press, Cambridge (2015)

31. Smirnov, N.V.: Estimate of deviation between empirical distribution functions in two independent samples. Bull. Moscow Univ. 2(2), 3–16 (1939)

32. Smith, M.J., Sala, C., Kanter, J.M., Veeramachaneni, K.: The machine learning bazaar: harnessing the ML ecosystem for effective system development. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, SIGMOD 2020, pp. 785–800. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3318464.3386146

33. Zhao, Y., Nasrullah, Z., Li, Z.: Pyod: a python toolbox for scalable outlier detection. J. Mach. Learn. Res. 20(96), 1–7 (2019). https://jmlr.org/papers/v20/19-011.html

34. Zhou, Y.: Rethinking reconstruction autoencoder-based out-of-distribution detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, pp. 7369–7377. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.00723