Repository: TimeEval/TimeEval (language: python)

Issue:
Use Bayesian Optimization for faster Hyper Parameter Optimization
Grid search for hyperparameter optimization can take too long as every combination of parameters is being tested, even the ones that are obviously bad. Using _Bayesian Optimization_ (BO), we could speed up the search because BO can find the best combination using multiple intelligent steps, exploring the search space.

**Refs**:

- https://machinelearningmastery.com/scikit-optimize-for-hyperparameter-tuning-in-machine-learning/
- http://hyperopt.github.io/hyperopt/


Discussion / hints:
Use Optuna: https://optuna.readthedocs.io/en/stable/index.html

Fix patch:
diff --git a/docs/api/index.rst b/docs/api/index.rst
index 4279e660..55ee5dc2 100644
--- a/docs/api/index.rst
+++ b/docs/api/index.rst
@@ -14,4 +14,5 @@ This section documents the public API of TimeEval.
    timeeval.metrics.thresholding
    timeeval.params
    timeeval.utils
-   timeeval.core
+   timeeval._core
+   integration/index
diff --git a/docs/api/integration/index.rst b/docs/api/integration/index.rst
new file mode 100644
index 00000000..435febad
--- /dev/null
+++ b/docs/api/integration/index.rst
@@ -0,0 +1,17 @@
+timeeval.integration package
+============================
+
+Available integration modules:
+
+.. toctree::
+   :maxdepth: 1
+
+   optuna
+
+Base class for TimeEval modules
+-------------------------------
+
+.. autoclass:: timeeval.integration.TimeEvalModule
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/api/integration/optuna.rst b/docs/api/integration/optuna.rst
new file mode 100644
index 00000000..616fc84c
--- /dev/null
+++ b/docs/api/integration/optuna.rst
@@ -0,0 +1,57 @@
+Optuna integration
+==================
+
+`Optuna <https://optuna.org>`_ is an automatic hyperparameter optimization framework and this integration allows you to
+use it within TimeEval. TimeEval will load the :class:`~timeeval.integration.optuna.OptunaModule` automatically if
+at least one algorithm uses :class:`~timeeval.params.BayesianParameterSearch` as its parameter search strategy. Please
+make sure that you install the required dependencies for Optuna before using this integration (we also recommend to
+install `psycopg2` to use the PostgreSQL storage backend):
+
+.. code-block:: bash
+
+   pip install 'optuna>=3.1.0' psycopg2
+
+The following Optuna features are supported:
+
+- Definition of search spaces using Optuna distributions for each algorithm (one study per algorithm):
+  :class:`~timeeval.params.BayesianParameterSearch`.
+- Configurable `samplers <https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/005_user_defined_sampler.html>`_.
+- Configurable `storage backends <https://optuna.readthedocs.io/en/stable/reference/storages.html>`_ (in-memory, RDB,
+  Journal, etc.).
+- `Resuming <https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/001_rdb.html>`_ of existing studies (via RDB
+  storage backend).
+- Parallel and distributed parameter search of a single or multiple studies (synchronized via RDB storage backend).
+
+
+.. warning::
+    Parameter search using the Optuna integration is **non-deterministic**. The results may vary between different runs,
+    even if the same seed is used (e.g., for the Optuna sampler or pruner). This is because TimeEval needs to re-seed
+    the Optuna samplers for every trial in distributed mode. This is necessary to ensure that initial random samples are
+    different over all workers.
+
+TimeEval will automatically manage an RDB storage backend if you use the default configuration. This allows you to start
+TimeEval in distributed mode and perform the parameter search in parallel and distributed.
+
+timeeval.integration.optuna.OptunaModule
+----------------------------------------
+
+.. autoclass:: timeeval.integration.optuna.OptunaModule
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+timeeval.integration.optuna.OptunaConfiguration
+-----------------------------------------------
+
+.. autoclass:: timeeval.integration.optuna.OptunaConfiguration
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+timeeval.integration.optuna.OptunaStudyConfiguration
+----------------------------------------------------
+
+.. autoclass:: timeeval.integration.optuna.OptunaStudyConfiguration
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/api/timeeval._core.rst b/docs/api/timeeval._core.rst
new file mode 100644
index 00000000..d5189ed5
--- /dev/null
+++ b/docs/api/timeeval._core.rst
@@ -0,0 +1,6 @@
+timeeval._core package
+======================
+
+.. warning::
+    This package contains TimeEval-internal classes and functions.
+    Do not use or change!
diff --git a/docs/api/timeeval.core.rst b/docs/api/timeeval.core.rst
deleted file mode 100644
index 2fc9e219..00000000
--- a/docs/api/timeeval.core.rst
+++ /dev/null
@@ -1,38 +0,0 @@
-timeeval.core package
-=====================
-
-.. warning::
-    This package contains TimeEval-internal classes and functions.
-    Do not use or change!
-
-timeeval.core.experiments.Experiment
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: timeeval.core.experiments.Experiment
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-timeeval.core.experiments.Experiments
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: timeeval.core.experiments.Experiments
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-timeeval.core.remote.Remote
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: timeeval.core.remote.Remote
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-timeeval.core.times.Times
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: timeeval.core.times.Times
-   :members:
-   :undoc-members:
-   :show-inheritance:
diff --git a/docs/api/timeeval.params.rst b/docs/api/timeeval.params.rst
index b5872901..8ab13edd 100644
--- a/docs/api/timeeval.params.rst
+++ b/docs/api/timeeval.params.rst
@@ -1,34 +1,42 @@
 timeeval.params package
 =======================
 
-timeeval.params.search.ParameterConfig
---------------------------------------
+timeeval.params.ParameterConfig
+-------------------------------
 
-.. autoclass:: timeeval.params.search.ParameterConfig
+.. autoclass:: timeeval.params.ParameterConfig
    :members:
    :undoc-members:
    :show-inheritance:
 
-timeeval.params.search.FixedParameters
---------------------------------------
+timeeval.params.FixedParameters
+-------------------------------
 
-.. autoclass:: timeeval.params.search.FixedParameters
+.. autoclass:: timeeval.params.FixedParameters
    :members:
    :undoc-members:
    :show-inheritance:
 
-timeeval.params.search.FullParameterGrid
+timeeval.params.FullParameterGrid
+---------------------------------
+
+.. autoclass:: timeeval.params.FullParameterGrid
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+timeeval.params.IndependentParameterGrid
 ----------------------------------------
 
-.. autoclass:: timeeval.params.search.FullParameterGrid
+.. autoclass:: timeeval.params.IndependentParameterGrid
    :members:
    :undoc-members:
    :show-inheritance:
 
-timeeval.params.search.IndependentParameterGrid
------------------------------------------------
+timeeval.params.BayesianParameterSearch
+---------------------------------------
 
-.. autoclass:: timeeval.params.search.IndependentParameterGrid
+.. autoclass:: timeeval.params.BayesianParameterSearch
    :members:
    :undoc-members:
    :show-inheritance:
diff --git a/docs/conf.py b/docs/conf.py
index d28360fe..7996f961 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -83,7 +83,8 @@
     'pandas': ('https://pandas.pydata.org/pandas-docs/version/1.3/', None),
     'sklearn': ('https://scikit-learn.org/0.24/', None),
     'statsmodels': ('https://www.statsmodels.org/v0.12.0/', None),
-    'pythresh': ('https://pythresh.readthedocs.io/en/latest', None)
+    'pythresh': ('https://pythresh.readthedocs.io/en/latest', None),
+    'optuna': ('https://optuna.readthedocs.io/en/v3.1.0/', None),
 }
 
 # -- sphinx_rtd_dark_mode settings -------------------------------------------
diff --git a/docs/requirements.txt b/docs/requirements.txt
index f75e6bf6..0cfe7028 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -2,3 +2,6 @@ sphinx==5.0.2
 myst-parser==0.18.0
 sphinx_rtd_theme==1.0.0
 sphinx_rtd_dark_mode==1.2.4
+
+# to build the optuna module autodoc API reference
+optuna
diff --git a/requirements.ci b/requirements.ci
index b645f8d3..f493c066 100644
--- a/requirements.ci
+++ b/requirements.ci
@@ -5,6 +5,8 @@ twine
 freezegun
 # for the test of the PyThreshThresholding class:
 pythresh>=0.2.8
+# for the test of the Optuna integration
+optuna>=3.1.0
 
 # typings
 types-requests
diff --git a/scripts/create_summary.py b/scripts/create_summary.py
index 0ceb8be4..9528c5e7 100644
--- a/scripts/create_summary.py
+++ b/scripts/create_summary.py
@@ -12,7 +12,7 @@
 from timeeval.adapters.docker import SCORES_FILE_NAME as DOCKER_SCORES_FILE_NAME
 from timeeval.constants import RESULTS_CSV, HYPER_PARAMETERS, METRICS_CSV, EXECUTION_LOG, ANOMALY_SCORES_TS
 from timeeval.data_types import ExecutionType
-from timeeval.core.experiments import Experiment as TimeEvalExperiment
+from timeeval._core.experiments import Experiment as TimeEvalExperiment
 from timeeval.heuristics import inject_heuristic_values
 from timeeval.utils.datasets import load_labels_only
 from timeeval_experiments.algorithm_configurator import AlgorithmConfigurator
diff --git a/scripts/reevaluate.py b/scripts/reevaluate.py
index b3ef395c..c2c8212f 100644
--- a/scripts/reevaluate.py
+++ b/scripts/reevaluate.py
@@ -12,7 +12,7 @@
 from timeeval.adapters.docker import SCORES_FILE_NAME as DOCKER_SCORES_FILE_NAME
 from timeeval.constants import RESULTS_CSV, HYPER_PARAMETERS, METRICS_CSV, ANOMALY_SCORES_TS
 from timeeval.data_types import ExecutionType
-from timeeval.core.experiments import Experiment as TimeEvalExperiment
+from timeeval._core.experiments import Experiment as TimeEvalExperiment
 from timeeval.metrics import FScoreAtK, PrecisionAtK
 from timeeval.utils.datasets import load_labels_only
 
diff --git a/setup.cfg b/setup.cfg
index d4d03f6d..9d3aebc6 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -19,56 +19,16 @@ skip_covered = True
 sort = Cover
 
 [mypy]
+exclude = (^timeeval_experiments/\d\d\d\d-\d\d-\d\d[_-].+$)
+ignore_missing_imports = True
 warn_unused_configs = True
 warn_unused_ignores = True
 warn_no_return = True
-exclude = (^timeeval_experiments/\d\d\d\d-\d\d-\d\d[_-].+$)
-
-[mypy-timeeval.*]
 warn_return_any = True
 disallow_untyped_defs = True
+disallow_incomplete_defs = True
+no_implicit_optional = True
+check_untyped_defs = True
 
-[mypy-sklearn.*]
-ignore_missing_imports = True
-
-[mypy-matplotlib.*]
-ignore_missing_imports = True
-
-[mypy-pandas.*]
-ignore_missing_imports = True
-
-[mypy-tqdm.*]
-ignore_missing_imports = True
-
-[mypy-dask.*]
-ignore_missing_imports = True
-
-[mypy-distributed.client.*]
-ignore_missing_imports = True
-
-[mypy-docker.*]
-ignore_missing_imports = True
-
-[mypy-durations.*]
-ignore_missing_imports = True
-
-[mypy-psutil.*]
-ignore_missing_imports = True
-
-[mypy-prts.*]
-ignore_missing_imports = True
-
-[mypy-statsmodels.*]
-ignore_missing_imports = True
-
-[mypy-joblib.*]
-ignore_missing_imports = True
-
-[mypy-numpyencoder.*]
-ignore_missing_imports = True
-
-[mypy-scipy.*]
-ignore_missing_imports = True
-
-[mypy-pythresh.*]
-ignore_missing_imports = True
+[mypy-optuna.samplers._tpe.*]
+follow_imports=skip
diff --git a/setup.py b/setup.py
index 6fef0806..88e0d94f 100644
--- a/setup.py
+++ b/setup.py
@@ -63,8 +63,8 @@ def run(self) -> None:
         import pytest
         from pytest import ExitCode
 
-        exit_code = pytest.main(["--cov-report=term", "--cov-report=xml:coverage.xml",
-                                 "--cov=timeeval", "--cov=timeeval_experiments.generator", "tests"])
+        exit_code = pytest.main(["--cov-report=term", "--cov-report=xml:coverage.xml", "--cov=timeeval",
+                                 "--cov=timeeval_experiments.generator", "--optuna", "tests"])
         if exit_code == ExitCode.TESTS_FAILED:
             raise ValueError("Tests failed!")
         elif exit_code == ExitCode.INTERRUPTED:
@@ -91,7 +91,7 @@ def finalize_options(self) -> None:
     def run(self) -> None:
         from mypy.main import main as mypy
 
-        args = ["--pretty", "timeeval", "timeeval_experiments", "tests"]
+        args = ["--pretty", "timeeval", "timeeval_experiments"]
         mypy(None, stdout=sys.stdout, stderr=sys.stderr, args=args)
 
 
diff --git a/timeeval/_core/__init__.py b/timeeval/_core/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/timeeval/core/experiments.py b/timeeval/_core/experiments.py
similarity index 69%
rename from timeeval/core/experiments.py
rename to timeeval/_core/experiments.py
index a7297aa3..a1ff529b 100644
--- a/timeeval/core/experiments.py
+++ b/timeeval/_core/experiments.py
@@ -1,23 +1,23 @@
 from contextlib import redirect_stdout
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Tuple, List, Generator, Optional
+from typing import Tuple, List, Optional, Iterator, Any, Dict
 
 import numpy as np
 import pandas as pd
 from sklearn.preprocessing import MinMaxScaler
 
+from .times import Times
 from ..algorithm import Algorithm
 from ..constants import EXECUTION_LOG, ANOMALY_SCORES_TS, METRICS_CSV, HYPER_PARAMETERS
-from ..core.times import Times
 from ..data_types import AlgorithmParameter, TrainingType, InputDimensionality
 from ..datasets import Datasets, Dataset
 from ..heuristics import inject_heuristic_values
 from ..metrics import Metric, DefaultMetrics
+from ..params import Params
 from ..resource_constraints import ResourceConstraints
 from ..utils.datasets import extract_features, load_dataset, load_labels_only
-from ..utils.encode_params import dump_params
-from ..utils.hash_dict import hash_dict
+from ..utils.encode_params import dump_params, dumps_params
 from ..utils.results_path import generate_experiment_path
 
 
@@ -25,7 +25,7 @@
 class Experiment:
     dataset: Dataset
     algorithm: Algorithm
-    params: dict
+    params: Params
     params_id: str
     repetition: int
     base_results_dir: Path
@@ -51,11 +51,11 @@ def results_path(self) -> Path:
         return generate_experiment_path(self.base_results_dir, self.algorithm.name, self.params_id,
                                         self.dataset_collection, self.dataset_name, self.repetition)
 
-    def build_args(self) -> dict:
+    def build_args(self) -> Dict[str, Any]:
         return {
             "results_path": self.results_path,
             "resource_constraints": self.resource_constraints,
-            "hyper_params": self.params,
+            "hyper_params": self.params.to_dict(),
             "dataset_details": self.dataset
         }
 
@@ -75,62 +75,75 @@ def scale_scores(y_true: np.ndarray, y_scores: np.ndarray) -> Tuple[np.ndarray,
             y_scores[~mask] = MinMaxScaler().fit_transform(scores).ravel()
         return y_true, y_scores
 
-    def evaluate(self) -> dict:
+    def evaluate(self) -> Dict[str, Any]:
         """
         Using TimeEval distributed, this method is executed on the remote node.
         """
-        with (self.results_path / EXECUTION_LOG).open("a") as logs_file:
-            print(f"Starting evaluation of experiment {self.name}\n=============================================\n",
-                  file=logs_file)
-
-        # persist hyper parameters to disk
+        # materialize and persist hyper parameters to disk
+        self.params = self.params.materialize()
         dump_params(self.params, self.results_path / HYPER_PARAMETERS)
-
-        # perform training if necessary
-        result = self._perform_training()
-
-        # perform execution
-        y_scores, execution_times = self._perform_execution()
-        result.update(execution_times)
-        # backup results to disk
-        pd.DataFrame([result]).to_csv(self.results_path / METRICS_CSV, index=False)
-
-        y_true = load_labels_only(self.resolved_test_dataset_path)
-        y_true, y_scores = self.scale_scores(y_true, y_scores)
-        # persist scores to disk
-        y_scores.tofile(str(self.results_path / ANOMALY_SCORES_TS), sep="\n")
-
-        with (self.results_path / EXECUTION_LOG).open("a") as logs_file:
-            print(f"Scoring algorithm {self.algorithm.name} with {','.join([m.name for m in self.metrics])} metrics",
-                  file=logs_file)
-
-            # calculate quality metrics
-            errors = 0
-            last_exception = None
-            for metric in self.metrics:
-                print(f"Calculating {metric.name}", file=logs_file)
-                try:
-                    score = metric(y_true, y_scores)
-                    result[metric.name] = score
-                    print(f"  = {score}", file=logs_file)
-                    logs_file.flush()
-                except Exception as e:
-                    print(f"Exception while computing metric {metric}: {e}", file=logs_file)
-                    errors += 1
-                    if str(e):
-                        last_exception = e
-                    continue
-
-        # write all results to disk (overwriting backup)
-        pd.DataFrame([result]).to_csv(self.results_path / METRICS_CSV, index=False)
-
-        # rethrow exception if no metric could be calculated
-        if errors == len(self.metrics) and last_exception is not None:
-            raise last_exception
-
+        hyper_params = dumps_params(self.params)  # must be loaded before assess() or fail() will be called!
+
+        try:
+            with (self.results_path / EXECUTION_LOG).open("a") as logs_file:
+                print(f"Starting evaluation of experiment {self.name}\n=============================================\n",
+                      file=logs_file)
+
+            # perform training if necessary
+            result = self._perform_training()
+
+            # perform execution
+            y_scores, execution_times = self._perform_execution()
+            result.update(execution_times)
+            # backup results to disk
+            pd.DataFrame([result]).to_csv(self.results_path / METRICS_CSV, index=False)
+
+            y_true = load_labels_only(self.resolved_test_dataset_path)
+            y_true, y_scores = self.scale_scores(y_true, y_scores)
+            # persist scores to disk
+            y_scores.tofile(str(self.results_path / ANOMALY_SCORES_TS), sep="\n")
+
+            with (self.results_path / EXECUTION_LOG).open("a") as logs_file:
+                print(f"Scoring algorithm {self.algorithm.name} with {','.join([m.name for m in self.metrics])} metrics",
+                      file=logs_file)
+
+                # calculate quality metrics
+                errors = 0
+                last_exception = None
+                for metric in self.metrics:
+                    print(f"Calculating {metric.name}", file=logs_file)
+                    try:
+                        score = metric(y_true, y_scores)
+                        result[metric.name] = score
+                        print(f"  = {score}", file=logs_file)
+                        logs_file.flush()
+                    except Exception as e:
+                        print(f"Exception while computing metric {metric}: {e}", file=logs_file)
+                        errors += 1
+                        if str(e):
+                            last_exception = e
+                        continue
+
+            # write all results to disk (overwriting backup)
+            pd.DataFrame([result]).to_csv(self.results_path / METRICS_CSV, index=False)
+
+            # potentially update parameter search space
+            self.params.assess(y_true, y_scores)
+
+            # rethrow exception if no metric could be calculated
+            if errors == len(self.metrics) and last_exception is not None:
+                raise last_exception
+
+        except Exception as e:
+            # on any exception, tell the parameter search process that this trial failed
+            self.params.fail()
+            raise e
+
+        # add hyperparameters to result
+        result["hyper_params"] = hyper_params
         return result
 
-    def _perform_training(self) -> dict:
+    def _perform_training(self) -> Dict[str, Any]:
         if self.algorithm.training_type == TrainingType.UNSUPERVISED:
             return {}
 
@@ -147,7 +160,7 @@ def _perform_training(self) -> dict:
             times = Times.from_train_algorithm(self.algorithm, X, self.build_args())
         return times.to_dict()
 
-    def _perform_execution(self) -> Tuple[np.ndarray, dict]:
+    def _perform_execution(self) -> Tuple[np.ndarray, Dict[str, Any]]:
         if self.algorithm.data_as_file:
             X: AlgorithmParameter = self.resolved_test_dataset_path
         else:
@@ -188,12 +201,13 @@ def __init__(self,
         self.force_training_type_match = force_training_type_match
         self.force_dimensionality_match = force_dimensionality_match
         self.experiment_combinations: Optional[pd.DataFrame] = pd.read_csv(experiment_combinations_file) if experiment_combinations_file else None
-        if self.skip_invalid_combinations:
+        if self.skip_invalid_combinations or self.experiment_combinations is not None:
             self._N: Optional[int] = None
         else:
             self._N = sum(
                 [len(algo.param_config) for algo in self.algorithms]
             ) * len(self.datasets) * self.repetitions
+        self._experiments: Optional[List[Experiment]] = None
 
     def _should_be_run(self, algorithm: Algorithm, dataset: Dataset, params_id: str) -> bool:
         return self.experiment_combinations is None or \
@@ -204,15 +218,15 @@ def _should_be_run(self, algorithm: Algorithm, dataset: Dataset, params_id: str)
                     (self.experiment_combinations.hyper_params_id == params_id)
                 ].empty
 
-    def __iter__(self) -> Generator[Experiment, None, None]:
+    def materialize_experiments(self) -> Iterator[Experiment]:
         for algorithm in self.algorithms:
-            for algorithm_config in algorithm.param_config:
-                for dataset in self.datasets:
-                    if self._check_compatible(dataset, algorithm):
+            for dataset in self.datasets:
+                if self._check_compatible(dataset, algorithm):
+                    for algorithm_config in algorithm.param_config.iter(algorithm, dataset):
                         test_path, train_path = self._resolve_dataset_paths(dataset, algorithm)
                         # create parameter hash before executing heuristics
                         # (they replace the parameter values, but we want to be able to group by original configuration)
-                        params_id = hash_dict(algorithm_config)
+                        params_id = algorithm_config.uid()
                         params = inject_heuristic_values(algorithm_config, algorithm, dataset, test_path)
                         if self._should_be_run(algorithm, dataset, params_id):
                             for repetition in range(1, self.repetitions + 1):
@@ -229,17 +243,18 @@ def __iter__(self) -> Generator[Experiment, None, None]:
                                     resolved_train_dataset_path=train_path
                                 )
 
+    def __iter__(self) -> Iterator[Experiment]:
+        if self._experiments is None:
+            self._experiments = list(self.materialize_experiments())
+        return iter(self._experiments)
+
     def __len__(self) -> int:
-        if self._N is None:
-            self._N = sum([
-                int(self._should_be_run(algorithm, dataset, hash_dict(algorithm_config)))
-                for algorithm in self.algorithms
-                for algorithm_config in algorithm.param_config
-                for dataset in self.datasets
-                if self._check_compatible(dataset, algorithm)
-                for _repetition in range(1, self.repetitions + 1)
-            ])
-        return self._N  # type: ignore
+        if self._N is None and self._experiments is not None:
+            self._N = len(self._experiments)
+        elif self._N is None:
+            self._experiments = list(self.materialize_experiments())
+            self._N = len(self._experiments)
+        return self._N
 
     def _resolve_dataset_paths(self, dataset: Dataset, algorithm: Algorithm) -> Tuple[Path, Optional[Path]]:
         test_dataset_path = self.dmgr.get_dataset_path(dataset.datasetId, train=False)
diff --git a/timeeval/core/remote.py b/timeeval/_core/remote.py
similarity index 87%
rename from timeeval/core/remote.py
rename to timeeval/_core/remote.py
index 24ca8c46..865a7f2d 100644
--- a/timeeval/core/remote.py
+++ b/timeeval/_core/remote.py
@@ -1,9 +1,11 @@
+from __future__ import annotations
+
 import logging
 import time
 from asyncio import Future, run_coroutine_threadsafe, get_event_loop
 from pathlib import Path
 from subprocess import Popen
-from typing import List, Callable, Tuple, Dict
+from typing import Any, List, Callable, Tuple, Dict
 
 import tqdm
 from dask import config as dask_config
@@ -22,7 +24,7 @@ def __init__(self, disable_progress_bar: bool = False,
         self.limits = resource_constraints
         self.log.debug(f"Remoting configuration: {self.config}\n"
                        f"with {self.limits.tasks_per_host} tasks per host")
-        self.futures: List[Future] = []
+        self.futures: List[Future[Dict[str, Any]]] = []
 
         # setup logging of Dask:
         self.log.info("Configuring dask logging")
@@ -68,7 +70,15 @@ def add_task(self, task: Callable, *args, **kwargs) -> Future:  # type: ignore[n
         self.futures.append(future)
         return future  # type: ignore
 
-    def run_on_all_hosts(self, tasks: List[Tuple[Callable, List, Dict]],
+    def run_on_scheduler(self, tasks: List[Tuple[Callable, List[Any], Dict[str, Any]]],
+                         msg: str = "Executing tasks on scheduler",
+                         progress: bool = True) -> None:
+        self.log.debug(f"Running {len(tasks)} tasks on scheduler")
+        for task, args, kwargs in tqdm.tqdm(tasks, desc=msg, disable=self.disable_progress_bar or not progress):
+            self.log.debug(f"({msg})Running task '{task}' with args {args} and kwargs {kwargs}")
+            self.client.run_on_scheduler(task, *args, **kwargs)
+
+    def run_on_all_hosts(self, tasks: List[Tuple[Callable, List[Any], Dict[str, Any]]],
                          msg: str = "Executing remote tasks",
                          progress: bool = True) -> None:
         self.log.debug(f"Running {len(tasks)} tasks on all cluster nodes and waiting for results")
diff --git a/timeeval/core/times.py b/timeeval/_core/times.py
similarity index 83%
rename from timeeval/core/times.py
rename to timeeval/_core/times.py
index 0dc5477a..c8a6b331 100644
--- a/timeeval/core/times.py
+++ b/timeeval/_core/times.py
@@ -1,6 +1,9 @@
+from __future__ import annotations
+
 import time
 from dataclasses import dataclass, asdict
-from typing import Optional, Dict, Callable, Tuple, Any, List
+from typing import Optional, Dict, Tuple, Any, List
+
 import numpy as np
 
 from ..algorithm import Algorithm
@@ -20,26 +23,26 @@ def result_keys() -> List[str]:
                  "execute_preprocess_time", "execute_main_time", "execute_postprocess_time"]
         return names
 
-    def to_dict(self) -> Dict:
+    def to_dict(self) -> Dict[str, Any]:
         dd = asdict(self)
         del dd["execution_type"]
         return {f"{self.execution_type.value}_{k}_time": v for k, v in dd.items()}
 
     @staticmethod
-    def from_execute_algorithm(algorithm: Algorithm, X: AlgorithmParameter, args: dict) -> Tuple[np.ndarray, 'Times']:
+    def from_execute_algorithm(algorithm: Algorithm, X: AlgorithmParameter, args: Dict[str, Any]) -> Tuple[np.ndarray, Times]:
         x, pre_time = timer(algorithm.preprocess, X, args) if algorithm.preprocess else (X, np.nan)
         x, main_time = timer(algorithm.execute, x, args)
         x, post_time = timer(algorithm.postprocess, x, args) if algorithm.postprocess else(x, np.nan)
         return x, Times(ExecutionType.EXECUTE, main_time, preprocess=pre_time, postprocess=post_time)
 
     @staticmethod
-    def from_train_algorithm(algorithm: Algorithm, X: AlgorithmParameter, args: dict) -> 'Times':
+    def from_train_algorithm(algorithm: Algorithm, X: AlgorithmParameter, args: Dict[str, Any]) -> Times:
         x, pre_time = timer(algorithm.preprocess, X, args) if algorithm.preprocess else (X, np.nan)
         x, main_time = timer(algorithm.train, x, args)
         return Times(ExecutionType.TRAIN, main_time, preprocess=pre_time)
 
 
-def timer(fn: Callable, X: AlgorithmParameter, args: dict) -> Tuple[Any, float]:
+def timer(fn: Any, X: AlgorithmParameter, args: Dict[str, Any]) -> Tuple[Any, float]:
     start = time.time()
     fn_result = fn(X, args)
     end = time.time()
diff --git a/timeeval/_version.py b/timeeval/_version.py
index d8584519..99ac92f6 100644
--- a/timeeval/_version.py
+++ b/timeeval/_version.py
@@ -1,2 +1,2 @@
-__version__: str = "1.2.10"
+__version__: str = "1.3.0rc1"
 """Version of this TimeEval installation"""
diff --git a/timeeval/adapters/base.py b/timeeval/adapters/base.py
index 05aa6ec9..691f1d7a 100644
--- a/timeeval/adapters/base.py
+++ b/timeeval/adapters/base.py
@@ -1,5 +1,5 @@
 from abc import ABC, abstractmethod
-from typing import Optional, Callable
+from typing import Optional, Callable, Any, Dict
 
 from ..data_types import AlgorithmParameter, ExecutionType
 
@@ -7,10 +7,10 @@
 class Adapter(ABC):
 
     @abstractmethod
-    def _call(self, dataset: AlgorithmParameter, args: dict) -> AlgorithmParameter:  # pragma: no cover
-        raise NotImplementedError("A subclass of Adapter must implement _call()!")
+    def _call(self, dataset: AlgorithmParameter, args: Dict[str, Any]) -> AlgorithmParameter:  # pragma: no cover
+        ...
 
-    def __call__(self, dataset: AlgorithmParameter, args: Optional[dict] = None) -> AlgorithmParameter:
+    def __call__(self, dataset: AlgorithmParameter, args: Optional[Dict[str, Any]] = None) -> AlgorithmParameter:
         args = args or {}
         if "executionType" not in args:
             args["executionType"] = ExecutionType.EXECUTE
diff --git a/timeeval/adapters/distributed.py b/timeeval/adapters/distributed.py
index ec920694..68b477da 100644
--- a/timeeval/adapters/distributed.py
+++ b/timeeval/adapters/distributed.py
@@ -1,7 +1,7 @@
 import getpass
 import logging
 import subprocess
-from typing import List
+from typing import List, Any, Dict
 
 from .base import Adapter
 from ..data_types import TSFunction, AlgorithmParameter
@@ -34,7 +34,7 @@ def _remote_command(self, remote_host: str) -> None:
             stdin.write(f"screen -dm bash -c \"{self.remote_command}\"")
             stdin.close()
 
-    def _call(self, dataset: AlgorithmParameter, args: dict) -> AlgorithmParameter:
+    def _call(self, dataset: AlgorithmParameter, args: Dict[str, Any]) -> AlgorithmParameter:
         # remote call
         for remote_host in self.remote_hosts:
             self._remote_command(remote_host)
diff --git a/timeeval/adapters/docker.py b/timeeval/adapters/docker.py
index d658004e..a923362b 100644
--- a/timeeval/adapters/docker.py
+++ b/timeeval/adapters/docker.py
@@ -16,6 +16,7 @@
 from ..data_types import ExecutionType
 from ..resource_constraints import ResourceConstraints, GB
 
+
 DATASET_TARGET_PATH = Path("/data")
 RESULTS_TARGET_PATH = Path("/results")
 SCORES_FILE_NAME = "docker-algorithm-scores.csv"
@@ -82,14 +83,14 @@ def _get_uid() -> str:
         else:
             return uid
 
-    def _get_compute_limits(self, args: dict) -> Tuple[int, float]:
+    def _get_compute_limits(self, args: Dict[str, Any]) -> Tuple[int, float]:
         limits: Tuple[int, float] = args.get("resource_constraints", ResourceConstraints()).get_compute_resource_limits(
             memory_overwrite=self.memory_limit,
             cpu_overwrite=self.cpu_limit
         )
         return limits
 
-    def _get_timeout(self, args: dict) -> Duration:
+    def _get_timeout(self, args: Dict[str, Any]) -> Duration:
         exec_type = args.get("executionType", "")
         constraints = args.get("resource_constraints", ResourceConstraints())
         if exec_type == ExecutionType.TRAIN or exec_type == ExecutionType.TRAIN.value:
@@ -98,14 +99,14 @@ def _get_timeout(self, args: dict) -> Duration:
             return constraints.get_execute_timeout(self.timeout)
 
     @staticmethod
-    def _should_use_prelim_model(args: dict) -> bool:
+    def _should_use_prelim_model(args: Dict[str, Any]) -> bool:
         exec_type = args.get("executionType", "")
         constraints = args.get("resource_constraints", ResourceConstraints())
         result: bool = (exec_type == ExecutionType.TRAIN or exec_type == ExecutionType.TRAIN.value) and constraints.use_preliminary_model_on_train_timeout
         return result
 
     @staticmethod
-    def _should_use_prelim_results(args: dict) -> bool:
+    def _should_use_prelim_results(args: Dict[str, Any]) -> bool:
         exec_type = args.get("executionType", "")
         constraints = args.get("resource_constraints", ResourceConstraints())
         result: bool = (exec_type == ExecutionType.EXECUTE or exec_type == ExecutionType.EXECUTE.value) and constraints.use_preliminary_scores_on_execute_timeout
@@ -118,7 +119,7 @@ def _results_path(args: Dict[str, Any], absolute: bool = False) -> Path:
             path = path.resolve()
         return path
 
-    def _run_container(self, dataset_path: Path, args: dict) -> Container:
+    def _run_container(self, dataset_path: Path, args: Dict[str, Any]) -> Container:
         client = docker.from_env()
 
         algorithm_interface = AlgorithmInterface(
@@ -158,7 +159,7 @@ def _run_container(self, dataset_path: Path, args: dict) -> Container:
             detach=True,
         )
 
-    def _run_until_timeout(self, container: Container, args: dict) -> None:
+    def _run_until_timeout(self, container: Container, args: Dict[str, Any]) -> None:
         timeout = self._get_timeout(args)
         try:
             result = container.wait(timeout=timeout.to_seconds())
@@ -207,13 +208,13 @@ def _run_until_timeout(self, container: Container, args: dict) -> None:
             print(f"Docker algorithm failed with status code '{result['StatusCode']}', consider container logs above.")
             raise DockerAlgorithmFailedError(f"Status '{result['StatusCode']}', please consider log files in {self._results_path(args, absolute=True)}!")
 
-    def _read_results(self, args: dict) -> np.ndarray:
+    def _read_results(self, args: Dict[str, Any]) -> np.ndarray:
         results: np.ndarray = np.genfromtxt(self._results_path(args, absolute=True) / SCORES_FILE_NAME, delimiter=",")
         return results
 
     # Adapter overwrites
 
-    def _call(self, dataset: AlgorithmParameter, args: dict) -> AlgorithmParameter:
+    def _call(self, dataset: AlgorithmParameter, args: Dict[str, Any]) -> AlgorithmParameter:
         assert isinstance(dataset, (WindowsPath, PosixPath)), \
             "Docker adapters cannot handle NumPy arrays! Please put in the path to the dataset."
         container = self._run_container(dataset, args)
diff --git a/timeeval/adapters/function.py b/timeeval/adapters/function.py
index 49b42c41..ec33e714 100644
--- a/timeeval/adapters/function.py
+++ b/timeeval/adapters/function.py
@@ -1,3 +1,5 @@
+from typing import Any, Dict
+
 from .base import Adapter
 from ..data_types import TSFunction, AlgorithmParameter
 
@@ -7,14 +9,14 @@ class FunctionAdapter(Adapter):
     def __init__(self, fn: TSFunction):
         self.fn = fn
 
-    def _call(self, dataset: AlgorithmParameter, args: dict) -> AlgorithmParameter:
+    def _call(self, dataset: AlgorithmParameter, args: Dict[str, Any]) -> AlgorithmParameter:
         # extract hyper parameters and forward them to the function
         params = args.get("hyper_params", {})
         return self.fn(dataset, params)
 
     @staticmethod
     def identity() -> 'FunctionAdapter':
-        def identity_fn(data: AlgorithmParameter, _: dict) -> AlgorithmParameter:
+        def identity_fn(data: AlgorithmParameter, _: Dict[str, Any]) -> AlgorithmParameter:
             return data
 
         return FunctionAdapter(fn=identity_fn)
diff --git a/timeeval/adapters/jar.py b/timeeval/adapters/jar.py
index 2b244935..c1ea97a8 100644
--- a/timeeval/adapters/jar.py
+++ b/timeeval/adapters/jar.py
@@ -1,4 +1,5 @@
 import subprocess
+from typing import Any, List, Dict
 
 import numpy as np
 
@@ -7,7 +8,7 @@
 
 
 class JarAdapter(Adapter):
-    def __init__(self, jar_file: str, output_file: str, args: list, kwargs: dict, verbose: bool = False) -> None:
+    def __init__(self, jar_file: str, output_file: str, args: List[Any], kwargs: Dict[str, Any], verbose: bool = False) -> None:
         self.jar_file = jar_file
         self.output_file = output_file
         self.args = args
@@ -24,7 +25,7 @@ def _read_results(self) -> np.ndarray:
         results: np.ndarray = np.loadtxt(self.output_file)
         return results
 
-    def _call(self, _1: AlgorithmParameter, _2: dict) -> np.ndarray:
+    def _call(self, _1: AlgorithmParameter, _2: Dict[str, Any]) -> np.ndarray:
         stdout = subprocess.STDOUT if self.verbose else subprocess.DEVNULL
         subprocess.call(f"java -jar {self.jar_file} {self._format_args()} {self._format_kwargs()}".split(),
                         stdout=stdout, stderr=subprocess.STDOUT)
diff --git a/timeeval/adapters/multivar.py b/timeeval/adapters/multivar.py
index 6e9c869f..bee5536e 100644
--- a/timeeval/adapters/multivar.py
+++ b/timeeval/adapters/multivar.py
@@ -1,6 +1,6 @@
 import multiprocessing as mp
 from enum import Enum
-from typing import Callable, List, Any
+from typing import Callable, List, Any, Dict
 
 import numpy as np
 
@@ -26,18 +26,18 @@ def __call__(self, data: List[np.ndarray]) -> np.ndarray:
 
 
 class MultivarAdapter(Adapter):
-    def __init__(self, fn: Callable[[np.ndarray, dict], np.ndarray], aggregation: AggregationMethod = AggregationMethod.MEAN,
+    def __init__(self, fn: Callable[[np.ndarray, Dict[str, Any]], np.ndarray], aggregation: AggregationMethod = AggregationMethod.MEAN,
                  n_jobs: int = 1) -> None:
         self.fn = fn
         self.aggregation = aggregation
         self.n_jobs = n_jobs
 
-    def _parallel_call(self, data: np.ndarray, args: dict) -> List[np.ndarray]:
+    def _parallel_call(self, data: np.ndarray, args: Dict[str, Any]) -> List[np.ndarray]:
         pool = mp.Pool(self.n_jobs)
         scores = pool.starmap(self.fn, [(data[:, c], args) for c in range(data.shape[1])])
         return scores
 
-    def _call(self, dataset: AlgorithmParameter, args: dict) -> np.ndarray:
+    def _call(self, dataset: AlgorithmParameter, args: Dict[str, Any]) -> np.ndarray:
         if isinstance(dataset, np.ndarray):
             if self.n_jobs > 1:
                 scores = self._parallel_call(dataset, args)
diff --git a/timeeval/algorithm.py b/timeeval/algorithm.py
index 0e0b6717..0b53d7e2 100644
--- a/timeeval/algorithm.py
+++ b/timeeval/algorithm.py
@@ -1,11 +1,17 @@
+from __future__ import annotations
+
 from dataclasses import dataclass, field
-from typing import Optional, Callable, Dict, Any
+from typing import TYPE_CHECKING
+
+from .data_types import ExecutionType, TrainingType, InputDimensionality
+from .params import ParameterConfig
+
 
-from timeeval.adapters.base import Adapter
-from timeeval.data_types import (
-    TSFunction, TSFunctionPost, ExecutionType, AlgorithmParameter, TrainingType, InputDimensionality
-)
-from timeeval.params import ParameterConfig
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from typing import Optional, Callable, Dict, Any
+    from .adapters.base import Adapter
+    from .data_types import TSFunction, TSFunctionPost, AlgorithmParameter
 
 
 @dataclass
@@ -42,7 +48,7 @@ class Algorithm:
                 },
             ]
 
-    param_config : timeeval.params.search.ParameterConfig
+    param_config : timeeval.params.ParameterConfig
         Optional object of type ParameterConfig to define a search grid or fixed parameters.
     training_type : timeeval.data_types.TrainingType
         Definition of training type to receive the correct dataset formats (needed if TimeEval is run with
@@ -72,7 +78,7 @@ class Algorithm:
     training_type: TrainingType = TrainingType.UNSUPERVISED
     input_dimensionality: InputDimensionality = InputDimensionality.UNIVARIATE
 
-    def train(self, dataset: AlgorithmParameter, args: Optional[dict] = None) -> AlgorithmParameter:
+    def train(self, dataset: AlgorithmParameter, args: Optional[Dict[str, Any]] = None) -> AlgorithmParameter:
         """Execute this algorithm's training procedure.
 
         .. warning::
@@ -114,7 +120,7 @@ def train(self, dataset: AlgorithmParameter, args: Optional[dict] = None) -> Alg
         args["executionType"] = ExecutionType.TRAIN
         return self.main(dataset, args)
 
-    def execute(self, dataset: AlgorithmParameter, args: Optional[dict] = None) -> AlgorithmParameter:
+    def execute(self, dataset: AlgorithmParameter, args: Optional[Dict[str, Any]] = None) -> AlgorithmParameter:
         """Execute this algorithm's test/execute procedure.
 
         .. warning::
diff --git a/timeeval/data_types.py b/timeeval/data_types.py
index 064f90c7..8df1e324 100644
--- a/timeeval/data_types.py
+++ b/timeeval/data_types.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 from enum import Enum
 from pathlib import Path
 from typing import Union, Callable
@@ -28,7 +30,7 @@ class TrainingType(Enum):
     """
 
     @staticmethod
-    def from_text(name: str) -> 'TrainingType':
+    def from_text(name: str) -> TrainingType:
         """Converts the string-representation to an Enum-object."""
         if name.lower() == TrainingType.SUPERVISED.value:
             return TrainingType.SUPERVISED
@@ -54,7 +56,7 @@ class InputDimensionality(Enum):
     """
 
     @staticmethod
-    def from_dimensions(n: int) -> 'InputDimensionality':
+    def from_dimensions(n: int) -> InputDimensionality:
         """Converts the feature/dimension/channel count to an Enum-object."""
         if n < 1:
             raise ValueError(f"Zero dimensional dataset is not supported!")
diff --git a/timeeval/datasets/analyzer.py b/timeeval/datasets/analyzer.py
index 47940476..a6a5d623 100755
--- a/timeeval/datasets/analyzer.py
+++ b/timeeval/datasets/analyzer.py
@@ -186,9 +186,9 @@ def _find_base_metadata(self) -> None:
         labels = self._df["is_anomaly"]
         label_groups = labels.groupby((labels.shift() != labels).cumsum())
         anomalies = [len(v) for k, v in label_groups if np.all(v)]
-        min_anomaly_length = np.min(anomalies) if anomalies else 0
+        min_anomaly_length = int(np.min(anomalies)) if anomalies else 0
         median_anomaly_length = int(np.median(anomalies)) if anomalies else 0
-        max_anomaly_length = np.max(anomalies) if anomalies else 0
+        max_anomaly_length = int(np.max(anomalies)) if anomalies else 0
         self.num_anomalies = len(anomalies)
         self.anomaly_length = AnomalyLength(
             min=min_anomaly_length,
diff --git a/timeeval/datasets/custom.py b/timeeval/datasets/custom.py
index 9f9b42d1..44c02a8f 100644
--- a/timeeval/datasets/custom.py
+++ b/timeeval/datasets/custom.py
@@ -1,7 +1,7 @@
 import json
 import warnings
 from pathlib import Path
-from typing import List, Union, Tuple, Optional, Dict, NamedTuple
+from typing import List, Union, Tuple, Optional, Dict, NamedTuple, Any
 
 import numpy as np
 import pandas as pd
@@ -64,13 +64,13 @@ def __init__(self, dataset_config: Union[str, Path]):
 
         self._dataset_store: Dict[str, CDEntry] = store
 
-    def _extract_path(self, obj: dict, key: str) -> Path:
+    def _extract_path(self, obj: Dict[str, Any], key: str) -> Path:
         path_string = obj[key]
         path: Path = self.root_path / path_string
         path = path.resolve()
         return path
 
-    def _validate_dataset(self, name: str, ds_obj: dict) -> None:
+    def _validate_dataset(self, name: str, ds_obj: Dict[str, Any]) -> None:
         if TEST_PATH_KEY not in ds_obj:
             raise ValueError(f"The dataset {name} misses the required '{TEST_PATH_KEY}' property.")
         elif not self._extract_path(ds_obj, TEST_PATH_KEY).exists():
@@ -78,7 +78,7 @@ def _validate_dataset(self, name: str, ds_obj: dict) -> None:
         if TRAIN_PATH_KEY in ds_obj and not self._extract_path(ds_obj, TRAIN_PATH_KEY).exists():
             raise ValueError(f"The train file for dataset {name} was not found (property '{TRAIN_PATH_KEY}')!")
 
-    def _analyze_dataset(self, name: str, ds_obj: dict) -> CDEntry:
+    def _analyze_dataset(self, name: str, ds_obj: Dict[str, Any]) -> CDEntry:
         dataset_id = _dataset_id(name)
         dataset_type = ds_obj.get(TYPE_KEY, "unknown")
         period = ds_obj.get(PERIOD_KEY, None)
diff --git a/timeeval/datasets/datasets.py b/timeeval/datasets/datasets.py
index 152eafa1..8868efbb 100644
--- a/timeeval/datasets/datasets.py
+++ b/timeeval/datasets/datasets.py
@@ -51,11 +51,11 @@ def __str__(self) -> str:
     @property
     @abc.abstractmethod
     def _log(self) -> logging.Logger:
-        raise NotImplementedError()
+        ...
 
     @abc.abstractmethod
     def _get_dataset_path_internal(self, dataset_id: DatasetId, train: bool = False) -> Path:
-        raise NotImplementedError()
+        ...
 
     def _create_index_file(self, filepath: Path) -> pd.DataFrame:
         df_temp = pd.DataFrame(
@@ -110,7 +110,7 @@ def safe_extract_path(name: str, train: bool) -> Optional[str]:
     @abc.abstractmethod
     def refresh(self, force: bool = False) -> None:
         """Re-read the benchmark dataset collection information from disk."""
-        raise NotImplementedError()
+        ...
 
     def get_collection_names(self) -> List[str]:
         """Returns the unique dataset collection names (includes custom datasets if present)."""
diff --git a/timeeval/datasets/metadata.py b/timeeval/datasets/metadata.py
index 845a9b35..08e0bccd 100644
--- a/timeeval/datasets/metadata.py
+++ b/timeeval/datasets/metadata.py
@@ -149,7 +149,7 @@ def default(self, o: Any) -> Any:
             return super(DatasetMetadataEncoder, self).default(o)
 
     @staticmethod
-    def object_hook(dct: dict) -> Any:
+    def object_hook(dct: Dict[str, Any]) -> Any:
         if "anomaly_length" in dct and "dataset_id" in dct and "is_train" in dct and "contamination" in dct:
             anomaly_length_dict = dct["anomaly_length"]
             anomaly_length = AnomalyLength(
diff --git a/timeeval/heuristics/AnomalyLengthHeuristic.py b/timeeval/heuristics/AnomalyLengthHeuristic.py
index 6a3ce4da..f6f69ebc 100755
--- a/timeeval/heuristics/AnomalyLengthHeuristic.py
+++ b/timeeval/heuristics/AnomalyLengthHeuristic.py
@@ -1,10 +1,17 @@
-from pathlib import Path
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
 from .base import TimeEvalParameterHeuristic
 
 
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from pathlib import Path
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+
+
 class AnomalyLengthHeuristic(TimeEvalParameterHeuristic):
     def __init__(self, agg_type: str = "median"):
         if agg_type not in ["min", "median", "max"]:
diff --git a/timeeval/heuristics/CleanStartSequenceSizeHeuristic.py b/timeeval/heuristics/CleanStartSequenceSizeHeuristic.py
index 4a99b4d5..11c77f39 100755
--- a/timeeval/heuristics/CleanStartSequenceSizeHeuristic.py
+++ b/timeeval/heuristics/CleanStartSequenceSizeHeuristic.py
@@ -1,11 +1,18 @@
-from pathlib import Path
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
 from timeeval.utils.datasets import load_labels_only
 from .base import TimeEvalParameterHeuristic
 
 
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from pathlib import Path
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+
+
 class CleanStartSequenceSizeHeuristic(TimeEvalParameterHeuristic):
     def __init__(self, max_factor: float = 0.1):
         self.max_factor = max_factor
diff --git a/timeeval/heuristics/ContaminationHeuristic.py b/timeeval/heuristics/ContaminationHeuristic.py
index 62eb2ea9..a8af7be3 100755
--- a/timeeval/heuristics/ContaminationHeuristic.py
+++ b/timeeval/heuristics/ContaminationHeuristic.py
@@ -1,13 +1,20 @@
-from pathlib import Path
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
 import numpy as np
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
 from .base import TimeEvalParameterHeuristic
 from ..utils.datasets import load_labels_only
 
 
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from pathlib import Path
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+
+
 class ContaminationHeuristic(TimeEvalParameterHeuristic):
     def __call__(self, algorithm: Algorithm, dataset_details: Dataset, dataset_path: Path, **kwargs) -> float:  # type: ignore[no-untyped-def]
         labels = load_labels_only(dataset_path)
diff --git a/timeeval/heuristics/DatasetIdHeuristic.py b/timeeval/heuristics/DatasetIdHeuristic.py
index e14e9c66..78416416 100755
--- a/timeeval/heuristics/DatasetIdHeuristic.py
+++ b/timeeval/heuristics/DatasetIdHeuristic.py
@@ -1,11 +1,18 @@
-from pathlib import Path
-from typing import Tuple
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
 from .base import TimeEvalParameterHeuristic
 
 
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from typing import Tuple
+    from pathlib import Path
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+
+
 class DatasetIdHeuristic(TimeEvalParameterHeuristic):
     """Heuristic to pass the dataset ID as a parameter value.
 
diff --git a/timeeval/heuristics/DefaultExponentialFactorHeuristic.py b/timeeval/heuristics/DefaultExponentialFactorHeuristic.py
index 6fa5a229..4b50db75 100755
--- a/timeeval/heuristics/DefaultExponentialFactorHeuristic.py
+++ b/timeeval/heuristics/DefaultExponentialFactorHeuristic.py
@@ -1,11 +1,18 @@
-from pathlib import Path
-from typing import Any
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
 from .base import TimeEvalParameterHeuristic
 
 
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from typing import Any
+    from pathlib import Path
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+
+
 class DefaultExponentialFactorHeuristic(TimeEvalParameterHeuristic):
     def __init__(self, exponent: int = 0, zero_fb: float = 1.0):
         if zero_fb == 0:
diff --git a/timeeval/heuristics/__init__.py b/timeeval/heuristics/__init__.py
index fbb195eb..65078a14 100755
--- a/timeeval/heuristics/__init__.py
+++ b/timeeval/heuristics/__init__.py
@@ -1,10 +1,10 @@
+from __future__ import annotations
+
 import re
 from copy import deepcopy
-from pathlib import Path
-from typing import Dict, Any
+from typing import TYPE_CHECKING, MutableMapping
 
-from timeeval import Algorithm
-from timeeval.datasets import Dataset
+from .base import TimeEvalParameterHeuristic
 from .AnomalyLengthHeuristic import AnomalyLengthHeuristic
 from .CleanStartSequenceSizeHeuristic import CleanStartSequenceSizeHeuristic
 from .ContaminationHeuristic import ContaminationHeuristic
@@ -15,7 +15,17 @@
 from .ParameterDependenceHeuristic import ParameterDependenceHeuristic
 from .PeriodSizeHeuristic import PeriodSizeHeuristic
 from .RelativeDatasetSizeHeuristic import RelativeDatasetSizeHeuristic
-from .base import TimeEvalParameterHeuristic
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from pathlib import Path
+    from typing import Any, TypeVar, Mapping
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+    from ..params import Params
+
+    T = TypeVar("T", Params, Mapping[str, Any], MutableMapping[str, Any])
 
 
 def _check_signature(signature: str) -> bool:
@@ -38,11 +48,16 @@ def TimeEvalHeuristic(signature: str) -> TimeEvalParameterHeuristic:
 
 
 def inject_heuristic_values(
-        params: Dict[str, Any],
+        params: T,
         algorithm: Algorithm,
         dataset_details: Dataset,
         dataset_path: Path,
-) -> Dict[str, Any]:
+) -> T:
+    # if not hasattr(params, "__setitem__") or not hasattr(params, "__delitem__"):
+    if not isinstance(params, MutableMapping):
+        # ignore all dynamic parameter search spaces that cannot be altered
+        return params
+
     updated_params = deepcopy(params)
     # defer dependence heuristics after all other heuristics
     heuristic_params = {(k, v) for k, v in params.items() if isinstance(v, str) and v.startswith("heuristic:")}
diff --git a/timeeval/heuristics/base.py b/timeeval/heuristics/base.py
index d711d460..a06492f5 100755
--- a/timeeval/heuristics/base.py
+++ b/timeeval/heuristics/base.py
@@ -14,7 +14,7 @@ def __call__(self, algorithm: Algorithm, dataset_details: Dataset, dataset_path:
         Calculate new parameter value based on information about algorithm, dataset metadata, and the dataset
         itself. If `None` is returned, the parameter is unset and the algorithm should use its default value.
         """
-        raise NotImplementedError
+        ...
 
     @property
     def name(self) -> str:
diff --git a/timeeval/integration/__init__.py b/timeeval/integration/__init__.py
new file mode 100644
index 00000000..1c313ff2
--- /dev/null
+++ b/timeeval/integration/__init__.py
@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from abc import ABC
+from typing import TYPE_CHECKING
+
+
+if TYPE_CHECKING:
+    from ..timeeval import TimeEval
+
+
+class TimeEvalModule(ABC):
+    """Base class for TimeEval modules that add additional functionality to TimeEval.
+
+    Inheriting classes can implement any of the following lifecycle-hooks:
+
+    1. :meth:`~timeeval.integration.TimeEvalModule.prepare`
+    2. :meth:`~timeeval.integration.TimeEvalModule.pre_run`
+    3. :meth:`~timeeval.integration.TimeEvalModule.post_run`
+    4. :meth:`~timeeval.integration.TimeEvalModule.finalize`
+
+    These methods are called at the corresponding time in the TimeEval run loop. Modules can assume that the TimeEval
+    configuration is already loaded and checked for user errors. If TimeEval is executed in distributed mode,
+    :attr:`timeeval.TimeEval.distributed` is set to ``True`` and remoting is already set up before the first call to
+    :meth:`~timeeval.integration.TimeEvalModule.prepare`.
+
+    .. note::
+        Implementing a TimeEval module is an advanced usage scenario and requires a good understanding of the internals
+        of TimeEval.
+    """
+    def prepare(self, timeeval: TimeEval) -> None:
+        """Called during the PREPARE-phase of TimeEval and before the individual algorithms' prepare-functions are
+        executed.
+
+        Parameters
+        ----------
+        timeeval : TimeEval
+            The TimeEval instance that is currently running.
+        """
+        pass
+
+    def pre_run(self, timeeval: TimeEval) -> None:
+        """Called before the EVALUATION-phase of TimeEval.
+
+        Parameters
+        ----------
+        timeeval : TimeEval
+            The TimeEval instance that is currently running.
+        """
+        pass
+
+    def post_run(self, timeeval: TimeEval) -> None:
+        """Called after the EVALUATION-phase of TimeEval.
+
+        Parameters
+        ----------
+        timeeval : TimeEval
+            The TimeEval instance that is currently running.
+        """
+        pass
+
+    def finalize(self, timeeval: TimeEval) -> None:
+        """Called during the FINALIZE-phase of TimeEval and after the individual algorithms' finalize-functions were
+        executed.
+
+        Parameters
+        ----------
+        timeeval : TimeEval
+            The TimeEval instance that is currently running.
+        """
+        pass
diff --git a/timeeval/integration/optuna/__init__.py b/timeeval/integration/optuna/__init__.py
new file mode 100644
index 00000000..486e3f80
--- /dev/null
+++ b/timeeval/integration/optuna/__init__.py
@@ -0,0 +1,3 @@
+from .config import OptunaConfiguration, OptunaStudyConfiguration
+from .module import OptunaModule
+from .params import OptunaLazyParams, OptunaParameterSearch
diff --git a/timeeval/integration/optuna/config.py b/timeeval/integration/optuna/config.py
new file mode 100644
index 00000000..a910ea05
--- /dev/null
+++ b/timeeval/integration/optuna/config.py
@@ -0,0 +1,168 @@
+from __future__ import annotations
+
+import logging
+
+import optuna.logging
+from dataclasses import dataclass
+from typing import Union, Optional, Any, Callable
+
+from optuna.pruners import BasePruner
+from optuna.samplers import BaseSampler
+from optuna.storages import BaseStorage
+from optuna.study import StudyDirection
+
+from timeeval import Metric
+
+
+@dataclass(init=True, repr=True)
+class OptunaConfiguration:
+    """Configuration options for the Optuna module. This includes default options for all Optuna studies.
+
+    Parameters
+    ----------
+    default_storage : str or Lambda returning instance of optuna.storages.BaseStorage
+        Storage to store and synchronize the results of the studies. Per default, TimeEval will use a journal file in
+        local execution mode and a PostgreSQL database in distributed execution mode. The database is automatically
+        started and stopped by TimeEval using the latest postgres-Docker image. Use ``"postgresql"`` to let TimeEval
+        handle starting and stopping a PostgreSQL database using Docker. Use ``"journal-file"`` to let TimeEval
+        create a local file as the storage backend. This only works in non-distributed mode.
+    default_sampler : optuna.samplers.BaseSampler, optional
+        Sampler to use for the study. If not provided, the default sampler is used.
+    default_pruner : optuna.pruners.BasePruner, optional
+        Pruner to use for the study. If not provided, the default pruner is used.
+    continue_existing_studies : bool, optional
+        If True, continue a study with the given name if it already exists in the storage backend. If False, raise an
+        error if a study with the same name already exists.
+    dashboard : bool, optional
+        If True, start the Optuna dashboard (within its own Docker container) to monitor the studies. In distributed
+        execution mode, the dashboard is started on the scheduler node.
+    remove_managed_containers : bool, optional
+        If True, remove the containers managed by TimeEval (e.g., the PostgreSQL database) when TimeEval is finished.
+    use_default_logging : bool, optional
+        If True, use the default logging configuration of the Optuna library. This will log the progress of the studies
+        to `stderr`. If False, use the logging configuration of TimeEval and propagates the Optuna log messages.
+    log_level : int or str, optional
+        The log level to use for the Optuna logger. The default is ``info`` = :attr:`logging.INFO` = ``20``.
+
+    See Also
+    --------
+    :func:`optuna.create_study`:
+        Used to create the Optuna study object; includes detailed explanation of the parameters.
+    :class:`timeeval.integration.optuna.OptunaModule`:
+        Optuna integration module for TimeEval.
+    """
+
+    default_storage: Union[str, Callable[[], BaseStorage]]
+    default_sampler: Optional[BaseSampler] = None
+    default_pruner: Optional[BasePruner] = None
+    continue_existing_studies: bool = False
+    dashboard: bool = False
+    remove_managed_containers: bool = False
+    use_default_logging: bool = False
+    log_level: Union[int, str] = logging.INFO
+
+    def __post_init__(self) -> None:
+        self._update_optuna_logging()
+
+    def __setattr__(self, key: str, value: Any) -> None:
+        super().__setattr__(key, value)
+        if key == "log_level" or key == "use_default_logging":
+            # update the Optuna logging configuration when changed
+            self._update_optuna_logging()
+
+    def _update_optuna_logging(self) -> None:
+        if self.use_default_logging:
+            optuna.logging.enable_default_handler()
+            optuna.logging.disable_propagation()
+        else:
+            optuna.logging.disable_default_handler()
+            optuna.logging.enable_propagation()
+        switcher = {
+            "debug": logging.DEBUG,
+            "info": logging.INFO,
+            "warning": logging.WARNING,
+            "error": logging.ERROR,
+            "critical": logging.CRITICAL,
+        }
+        level = switcher[self.log_level] if isinstance(self.log_level, str) else self.log_level
+        optuna.logging.set_verbosity(level)
+
+    @staticmethod
+    def default(distributed: bool) -> OptunaConfiguration:
+        if distributed:
+            return OptunaConfiguration(default_storage="postgresql")
+        else:
+            return OptunaConfiguration(default_storage="journal-file")
+
+
+@dataclass(init=True, repr=True, frozen=True)
+class OptunaStudyConfiguration:
+    """Configuration for :class:`~timeeval.params.BayesianParameterSearch`.
+
+    The parameters ``n_trials`` and ``metric`` are required. All other parameters are optional and will be filled with
+    the default values from the global Optuna configuration if not provided.
+
+    Parameters
+    ----------
+    n_trials : int
+        Number of trials to perform.
+    metric : Metric
+        TimeEval metric to use as the studies objective function.
+    storage : str or Lambda returning instance of optuna.storages.BaseStorage, optional
+        Storage to store the results of the study.
+    sampler : optuna.samplers.BaseSampler, optional
+        Sampler to use for the study. If not provided, the default sampler is used.
+    pruner : optuna.pruners.BasePruner, optional
+        Pruner to use for the study. If not provided, the default pruner is used.
+    direction : str or optuna.study.StudyDirection, optional
+        Direction of the optimization (minimize or `maximize`). If ``None``, the Optuna default direction is used.
+    continue_existing_study : bool, optional
+        If True, continue a study with the given name if it already exists in the storage backend. If False, raise an
+        error if a study with the same name already exists.
+
+    See Also
+    --------
+    :func:`optuna.create_study`:
+        Used to create the Optuna study object; includes detailed explanation of the parameters.
+    :class:`timeeval.integration.optuna.OptunaModule`:
+        Optuna integration module for TimeEval.
+    """
+
+    n_trials: int
+    metric: Metric
+    storage: Optional[Union[str, Callable[[], BaseStorage]]] = None
+    sampler: Optional[BaseSampler] = None
+    pruner: Optional[BasePruner] = None
+    direction: Optional[Union[str, StudyDirection]] = "maximize"
+    continue_existing_study: bool = False
+
+    def update_unset_options(self, global_config: OptunaConfiguration) -> OptunaStudyConfiguration:
+        return OptunaStudyConfiguration(
+            n_trials=self.n_trials,
+            metric=self.metric,
+            storage=self.storage or global_config.default_storage,
+            sampler=self.sampler or global_config.default_sampler,
+            pruner=self.pruner or global_config.default_pruner,
+            direction=self.direction,
+            continue_existing_study=self.continue_existing_study or global_config.continue_existing_studies,
+        )
+
+    def copy(self,
+             n_trials: Optional[int] = None,
+             metric: Optional[Metric] = None,
+             storage: Optional[Union[str, Callable[[], BaseStorage]]] = None,
+             sampler: Optional[BaseSampler] = None,
+             pruner: Optional[BasePruner] = None,
+             direction: Optional[Union[str, StudyDirection]] = None,
+             continue_existing_study: Optional[bool] = None) -> OptunaStudyConfiguration:
+        """Create a copy of this configuration with the given parameters replaced."""
+
+        return OptunaStudyConfiguration(
+            n_trials=n_trials or self.n_trials,
+            metric=metric or self.metric,
+            storage=storage or self.storage,
+            sampler=sampler or self.sampler,
+            pruner=pruner or self.pruner,
+            direction=direction or self.direction,
+            continue_existing_study=continue_existing_study or self.continue_existing_study,
+        )
diff --git a/timeeval/integration/optuna/module.py b/timeeval/integration/optuna/module.py
new file mode 100644
index 00000000..d3c1e5e1
--- /dev/null
+++ b/timeeval/integration/optuna/module.py
@@ -0,0 +1,217 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+import socket
+from typing import TYPE_CHECKING, Optional, List, Tuple, Callable, Any, Dict
+
+import docker
+import optuna.storages
+from docker.errors import DockerException, NotFound
+from optuna.storages import JournalStorage, JournalFileStorage, JournalFileOpenLock, RDBStorage
+
+from timeeval.integration import TimeEvalModule
+
+
+if TYPE_CHECKING:
+    from .config import OptunaConfiguration
+    from ...timeeval import TimeEval
+    from distributed import Scheduler
+    from optuna.study import StudySummary
+
+
+POSTGRESQL_IMAGE_NAME = "postgres:latest"
+OPTUNA_DASHBOARD_IMAGE_NAME = "ghcr.io/optuna/optuna-dashboard:v0.8.1"
+DB_CONTAINER_NAME = "timeeval-optuna-db"
+DB_MAX_CONNECTIONS = 1000
+DB_STARTUP_DELAY = 5  # in seconds
+DASHBOARD_CONTAINER_NAME = "timeeval-optuna-dashboard"
+log = logging.getLogger("OptunaModule")
+
+
+# use an async function to not block the scheduler while waiting for the database to start up
+async def _start_postgres_container(scheduler: Optional[Scheduler] = None, password: str = "postgres",
+                                    port: int = 5432) -> None:
+    import docker
+    client = docker.from_env()
+    log.debug(f"Starting postgres container on port {port}")
+    client.containers.run(
+        POSTGRESQL_IMAGE_NAME,
+        f"-c max_connections={DB_MAX_CONNECTIONS}",
+        name=DB_CONTAINER_NAME,
+        environment={
+            "POSTGRES_PASSWORD": password,
+        },
+        ports={"5432/tcp": port},
+        detach=True,
+    )
+    log.debug(f"Waiting {DB_STARTUP_DELAY} seconds for database to start up")
+    await asyncio.sleep(DB_STARTUP_DELAY)
+
+
+def _start_dashboard_container(scheduler: Optional[Scheduler] = None,
+                               storage: str = "postgresql://postgres:postgres@localhost:5432/postgres") -> None:
+    import docker
+    client = docker.from_env()
+    log.debug("Starting dashboard container")
+    client.containers.run(
+        OPTUNA_DASHBOARD_IMAGE_NAME,
+        storage,
+        name=DASHBOARD_CONTAINER_NAME,
+        network_mode="host",
+        # ports={"8080/tcp": 8080},
+        # extra_hosts={"host.docker.internal": "host-gateway"},
+        detach=True,
+    )
+
+
+def _stop_containers(scheduler: Optional[Scheduler] = None, remove: bool = False) -> None:
+    import docker
+    client = docker.from_env()
+    try:
+        c = client.containers.get(DASHBOARD_CONTAINER_NAME)
+        log.debug("Stopping dashboard container")
+        c.stop()
+        if remove:
+            c.remove(v=True, force=True)
+    except NotFound:
+        pass
+    try:
+        c = client.containers.get(DB_CONTAINER_NAME)
+        log.debug("Stopping database container")
+        c.stop()
+        if remove:
+            c.remove(v=True, force=True)
+    except NotFound:
+        pass
+
+
+class OptunaModule(TimeEvalModule):
+    """This module is automatically loaded when at least one algorithm uses
+    :class:`timeeval.params.BayesianParameterSearch` as parameter config.
+
+    TimeEval provides the option to use an automatically managed PostgreSQL database as the storage backend for the
+    Optuna studies. The database is started as an additional Docker container either on the local machine or on the
+    scheduler node in distributed execution mode. The database is automatically stopped when TimeEval is finished. The
+    database storage backend allows you to monitor the studies using the Optuna dashboard (that can also be started
+    automatically using another Docker container) and the distributed execution of the studies. This is the default
+    behavior if no storage backend is specified in the configuration.
+
+    Parameters
+    ----------
+    config : OptunaConfiguration
+        The configuration for the Optuna module.
+    """
+    def __init__(self, config: OptunaConfiguration):
+        self.config = config
+        # check configuration:
+        check_docker = ""
+        if self.config.dashboard:
+            check_docker = "start the Optuna dashboard"
+        if self.config.default_storage is None:
+            raise ValueError("No default storage specified!")
+        elif isinstance(self.config.default_storage, str) and self.config.default_storage == "postgresql":
+            and_required = " and " if check_docker else ""
+            check_docker += and_required + f"start the Optuna storage {self.config.default_storage}"
+        self._check_docker_available(check_docker)
+
+        self.storage_url: Optional[str] = None
+
+    @staticmethod
+    def _check_docker_available(reason: str) -> None:
+        if reason:
+            try:
+                docker.from_env()
+            except DockerException as e:
+                raise ValueError(f"No docker client found, but docker is required to {reason}!") from e
+
+    def prepare(self, timeeval: TimeEval) -> None:
+        log.info("Optuna module: preparing ...")
+        tasks: List[Tuple[Callable, List[Any], Dict[str, Any]]] = []
+
+        if isinstance(self.config.default_storage, str) and self.config.default_storage == "postgresql":
+            host = timeeval.remote.config.scheduler_host if timeeval.distributed else socket.gethostname()
+            port = 5432
+            password = "hairy_bumblebee"
+            # set actual connection string:
+            storage_url = f"postgresql://postgres:{password}@{host}:{port}/postgres"
+            self.config.default_storage = lambda: RDBStorage(
+                storage_url,
+                engine_kwargs={"pool_size": 1, "max_overflow": 2, "pool_timeout": 60}
+            )
+            log.debug(f"starting managed postgresql storage backend at ({storage_url})...")
+            self.storage_url = storage_url
+
+            tasks.append((_start_postgres_container, [], {"password": password, "port": port}))
+
+        elif isinstance(self.config.default_storage, str) and self.config.default_storage == "journal-file":
+            journal_file_path = str(timeeval.results_path / "optuna-journal.log")
+            self.config.default_storage = lambda: JournalStorage(
+                JournalFileStorage(journal_file_path, lock_obj=JournalFileOpenLock(journal_file_path))
+            )
+
+        if self.config.dashboard:
+            if self.storage_url is None and not isinstance(self.config.default_storage, str):
+                # user provided a custom storage backend, try to get the URL:
+                from sqlalchemy.exc import OperationalError
+                try:
+                    storage = optuna.storages.get_storage(self.config.default_storage())
+                    if hasattr(storage, "url"):
+                        self.storage_url = storage.url  # type: ignore
+                    else:
+                        log.warning(f"Could not find dashboard connection URL for storage {self.config.default_storage}, "
+                                    "not starting dashboard!")
+                except OperationalError as e:
+                    log.warning("Could not find connection URL for storage, not starting dashboard!", exc_info=e)
+
+            if self.storage_url is not None:
+                storage_host = timeeval.remote.config.scheduler_host if timeeval.distributed else "localhost"
+                dashboard_url = f"http://{storage_host}:8080"
+                log.debug(f"starting managed Optuna dashboard ({dashboard_url})...")
+                print(f"\n\tOpen Optuna dashboard at {dashboard_url}\n")
+                tasks.append((_start_dashboard_container, [], {"storage": self.storage_url}))
+
+        if timeeval.distributed:
+            log.debug(f"distributed mode: starting {len(tasks)} containers on scheduler")
+            timeeval.remote.run_on_scheduler(tasks, msg="Starting Optuna containers on scheduler")
+        else:
+            for task, args, kwargs in tasks:
+                task(*args, **kwargs)
+
+        # update optuna study configurations:
+        log.debug("updating optuna study configurations ...")
+        for algo in timeeval.exps.algorithms:
+            if hasattr(algo.param_config, "update_config"):
+                algo.param_config.update_config(self.config)  # type: ignore
+        log.info("Optuna module: preparing done.")
+
+    def finalize(self, timeeval: TimeEval) -> None:
+        log.info("Optuna module: finalizing ...")
+        if timeeval.distributed:
+            log.debug("distributed mode: stopping containers on scheduler")
+            timeeval.remote.run_on_scheduler(
+                [(_stop_containers, [], {"remove": self.config.remove_managed_containers})],
+                msg="Stopping Optuna containers on scheduler"
+            )
+        else:
+            _stop_containers(remove=self.config.remove_managed_containers)
+        log.info("Optuna module: finalizing done.")
+
+    def load_studies(self) -> List[StudySummary]:
+        """Load all studies from the default storage. This does not include studies, which were stored in a different
+        storage backend (i.a. where the storage backend was changed using the
+        :class:`timeeval.integration.optuna.OptunaStudyConfiguration`).
+
+        Returns
+        -------
+        study_summaries : List[StudySummary]
+            A list of study summaries.
+
+        See Also
+        --------
+        :func:`optuna.study.get_all_study_summaries` : Optuna function which is used to load the studies.
+        :class:`optuna.study.StudySummary` : Optuna class which is used to represent the study summaries.
+        """
+        return optuna.get_all_study_summaries(
+            self.config.default_storage if isinstance(self.config.default_storage, str) else self.config.default_storage()
+        )
diff --git a/timeeval/integration/optuna/params.py b/timeeval/integration/optuna/params.py
new file mode 100644
index 00000000..1ace8703
--- /dev/null
+++ b/timeeval/integration/optuna/params.py
@@ -0,0 +1,201 @@
+from __future__ import annotations
+
+import socket
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+import numpy as np
+import optuna
+from optuna.storages import BaseStorage, RDBStorage
+from optuna.trial import TrialState
+
+from timeeval.params import ParameterConfig, Params
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from typing import Iterator, Any, Mapping, Dict, Optional, ItemsView, Union
+    from optuna import Study, Trial
+    from optuna.distributions import BaseDistribution
+    from timeeval import Algorithm
+    from timeeval.datasets import Dataset
+    from .config import OptunaConfiguration, OptunaStudyConfiguration
+
+
+@dataclass(init=False, repr=True)
+class OptunaLazyParams(Params):
+    study_name: str
+    index: int
+    distributions: Dict[str, BaseDistribution]
+    config: OptunaStudyConfiguration
+
+    def __init__(self,
+                 study_name: str,
+                 index: int,
+                 distributions: Mapping[str, BaseDistribution],
+                 config: OptunaStudyConfiguration,
+                 ):
+        super().__init__()
+        self.study_name = study_name
+        self.index = index
+        self.distributions = dict(distributions)
+        self.config = config
+        self._study: Optional[Study] = None
+        self._trial: Optional[Trial] = None
+
+    def __len__(self) -> int:
+        return len(self.distributions)
+
+    def __iter__(self) -> Iterator[str]:
+        return iter(self.distributions)
+
+    def __getitem__(self, param_name: str) -> Any:
+        return self.trial().params[param_name]
+
+    def trial(self) -> Trial:
+        if self._trial is None:
+            raise ValueError("The parameters have not yet been materialized!")
+        return self._trial
+
+    def study(self) -> Study:
+        if self._study is None:
+            raise ValueError("The parameters have not yet been materialized!")
+        return self._study
+
+    def items(self) -> ItemsView[str, Any]:
+        return dict((k, self.uid()) for k in self.distributions).items()
+
+    def materialize(self) -> Params:
+        # only materialize once:
+        if self._study is None:
+            if self.config.storage is None or isinstance(self.config.storage, str):
+                storage: Union[None, str, BaseStorage] = self.config.storage
+            else:
+                storage = self.config.storage()
+            self._study = optuna.load_study(
+                study_name=self.study_name,
+                storage=storage,
+                sampler=self.config.sampler,
+                pruner=self.config.pruner,
+            )
+            # we need to manually reseed the sampler because we are not in an optimize()-call:
+            self._study.sampler.reseed_rng()
+
+        if self._trial is None:
+            self._trial = self._study.ask(self.distributions)
+            self._trial.set_user_attr("uid", self.uid())
+            self._trial.set_user_attr("node", socket.gethostname())
+        return self
+
+    def assess(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
+        t = self.trial()
+        score = self.config.metric(y_true, y_score)
+        self.study().tell(t, score)
+        self.close()
+        return score
+
+    def fail(self) -> None:
+        self.study().tell(self.trial(), state=TrialState.FAIL)
+        self.close()
+
+    def uid(self) -> str:
+        return self.build_uid(self.study_name, self.index)
+
+    def to_dict(self) -> Dict[str, Any]:
+        params: Dict[str, Any] = self.trial().params
+        return params
+
+    def close(self) -> None:
+        if self._trial is not None:
+            del self._trial
+        if self._study is not None:
+            storage = self._study._storage
+            if isinstance(storage, RDBStorage):
+                storage.remove_session()
+            del self._study
+
+    @staticmethod
+    def build_uid(study_name: str, i: int) -> str:
+        return f"{study_name}-{i}"
+
+
+class OptunaParameterSearch(ParameterConfig):
+    """Implementation of the Bayesian optimization using Optuna library."""
+
+    def __init__(self, config: OptunaStudyConfiguration,
+                 params: Mapping[str, BaseDistribution],
+                 include_default_params: bool = False):
+        self._config = config
+        self._distributions = params
+        self._include_default_params = include_default_params
+
+    def iter(self, algorithm: Algorithm, dataset: Dataset) -> Iterator[Params]:
+        # create the study and enforce a common name for all trials of the study (this will create the study in the
+        # storage backend so that it can be accessed by all workers):
+        if self._config.storage is None or isinstance(self._config.storage, str):
+            storage: Union[None, str, BaseStorage] = self._config.storage
+        else:
+            storage = self._config.storage()
+        study = optuna.create_study(
+            study_name=f"{algorithm.name}-{dataset.name}",
+            storage=storage,
+            sampler=self._config.sampler,
+            pruner=self._config.pruner,
+            direction=self._config.direction,
+            load_if_exists=self._config.continue_existing_study,
+        )
+        study.set_user_attr("algorithm", algorithm.name)
+        study.set_user_attr("dataset", dataset.name)
+        study.set_user_attr("metric", self._config.metric.name)
+        study.set_user_attr("direction", str(self._config.direction).lower())
+        study.set_user_attr("includes_default_params", self._include_default_params)
+        study_name = study.study_name
+
+        if self._include_default_params:
+            try:
+                study.enqueue_trial(self._default_params(algorithm))
+                yield OptunaLazyParams(study_name, 0, self._distributions, self._config)
+            except ValueError as e:
+                from .module import log
+                log.warning(f"Could not create default parameters for {algorithm.name}, skipping!", exc_info=e)
+
+        # cleanup (each trial manages their own DB connection)
+        del study
+        if isinstance(storage, RDBStorage):
+            storage.remove_session()
+
+        for i in range(self._config.n_trials):
+            yield OptunaLazyParams(study_name, i + self._include_default_params, self._distributions, self._config)
+
+    def __len__(self) -> int:
+        return self._config.n_trials + int(self._include_default_params)
+
+    def update_config(self, global_config: OptunaConfiguration) -> None:
+        self._config = self._config.update_unset_options(global_config)
+
+    def _default_params(self, algorithm: Algorithm) -> Dict[str, Any]:
+        params = {}
+        for param_name in self._distributions:
+            if param_name not in algorithm.param_schema:
+                continue
+            schema = algorithm.param_schema[param_name]
+            value = schema["defaultValue"]
+            tpe = schema["type"].lower()
+
+            if value is None or "enum" in tpe:
+                params[param_name] = value
+            elif tpe == "int":
+                params[param_name] = int(value)
+            elif tpe == "float":
+                params[param_name] = float(value)
+            elif tpe == "boolean":
+                params[param_name] = bool(value)
+            elif tpe == "str":
+                params[param_name] = str(value)
+            else:
+                raise ValueError(f"Unsupported parameter type {tpe} for param {param_name} in {algorithm.name}!")
+
+        if len(params) == 0:
+            raise ValueError(f"No default parameters found for {algorithm.name}!")
+
+        return params
diff --git a/timeeval/metrics/auc_metrics.py b/timeeval/metrics/auc_metrics.py
index c9830b13..3f70786d 100644
--- a/timeeval/metrics/auc_metrics.py
+++ b/timeeval/metrics/auc_metrics.py
@@ -1,5 +1,5 @@
 from abc import ABC
-from typing import Iterable, Callable
+from typing import Iterable, Callable, Any
 
 import numpy as np
 from sklearn.metrics import auc, roc_curve, precision_recall_curve
@@ -17,16 +17,19 @@ def __init__(self, plot: bool = False, plot_store: bool = False) -> None:
         self._plot = plot
         self._plot_store = plot_store
 
-    def _auc(self, y_true: np.ndarray, y_score: Iterable[float], _curve_function: Callable) -> float:
-        x, y, thresholds = _curve_function(y_true, y_score)
-        if "precision_recall" in _curve_function.__name__:
+    def _auc(self,
+             y_true: np.ndarray,
+             y_score: Iterable[float],
+             curve_function: Callable[[np.ndarray, np.ndarray], Any]) -> float:
+        x, y, thresholds = curve_function(y_true, np.array(y_score))
+        if "precision_recall" in curve_function.__name__:
             # swap x and y
             x, y = y, x
         area: float = auc(x, y)
         if self._plot:
             import matplotlib.pyplot as plt
 
-            name = _curve_function.__name__
+            name = curve_function.__name__
             plt.plot(x, y, label=name, drawstyle="steps-post")
             # plt.plot([0, 1], [0, 1], linestyle="--", label="Random")
             plt.title(f"{name} | area = {area:.4f}")
diff --git a/timeeval/metrics/classification_metrics.py b/timeeval/metrics/classification_metrics.py
index 5221ca50..9412b344 100644
--- a/timeeval/metrics/classification_metrics.py
+++ b/timeeval/metrics/classification_metrics.py
@@ -1,4 +1,4 @@
-import abc
+from abc import ABC, abstractmethod
 
 import numpy as np
 from sklearn.metrics import precision_score, recall_score, f1_score
@@ -7,7 +7,7 @@
 from .thresholding import ThresholdingStrategy, NoThresholding
 
 
-class ClassificationMetric(Metric, abc.ABC):
+class ClassificationMetric(Metric, ABC):
     """Base class for standard classification metrics.
 
     All classification metrics are defined over binary classification predictions (zeros or ones), thus all of them
@@ -25,10 +25,10 @@ def score(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
     def supports_continuous_scorings(self) -> bool:
         return not isinstance(self._thresholding_strategy, NoThresholding)
 
-    @abc.abstractmethod
+    @abstractmethod
     def _internal_score(self, y_true: np.ndarray, y_pred: np.ndarray) -> float:
         """Classification metrics should implement this method to compute the metric score."""
-        pass
+        ...
 
 
 class Precision(ClassificationMetric):
diff --git a/timeeval/metrics/metric.py b/timeeval/metrics/metric.py
index a8eda2a2..0fb1c16d 100644
--- a/timeeval/metrics/metric.py
+++ b/timeeval/metrics/metric.py
@@ -1,12 +1,12 @@
-import abc
 import warnings
+from abc import ABC, abstractmethod
 from typing import Tuple
 
 import numpy as np
 from sklearn.utils import column_or_1d, assert_all_finite, check_consistent_length
 
 
-class Metric(abc.ABC):
+class Metric(ABC):
     """Base class for metric implementations that score anomaly scorings against ground truth binary labels. Every
     subclass must implement :func:`~timeeval.metrics.Metric.name`, :func:`~timeeval.metrics.Metric.score`, and
     :func:`~timeeval.metrics.Metric.supports_continuous_scorings`.
@@ -96,17 +96,17 @@ def _validate_scores(self, y_true: np.ndarray, y_score: np.ndarray,
         return y_true, y_score
 
     @property
-    @abc.abstractmethod
+    @abstractmethod
     def name(self) -> str:
         """Returns the unique name of this metric."""
         ...
 
-    @abc.abstractmethod
+    @abstractmethod
     def score(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
         """Implementation of the metric's scoring function."""
         ...
 
-    @abc.abstractmethod
+    @abstractmethod
     def supports_continuous_scorings(self) -> bool:
         """Whether this metric accepts continuous anomaly scorings as input (``True``) or binary classification
         labels (``False``)."""
diff --git a/timeeval/metrics/range_metrics.py b/timeeval/metrics/range_metrics.py
index 964a4f54..042e318c 100644
--- a/timeeval/metrics/range_metrics.py
+++ b/timeeval/metrics/range_metrics.py
@@ -198,8 +198,9 @@ def __init__(self, max_samples: int = 50, r_alpha: float = 0.5, p_alpha: float =
     def score(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
         return self._auc(y_true, y_score, self._range_precision_recall_curve)
 
-    def _range_precision_recall_curve(self, y_true: np.ndarray, y_score: np.ndarray) -> Tuple[
-        np.ndarray, np.ndarray, np.ndarray]:
+    def _range_precision_recall_curve(self,
+                                      y_true: np.ndarray,
+                                      y_score: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
         thresholds = np.unique(y_score)
         thresholds.sort()
         # The first precision and recall values are precision=class balance and recall=1.0, which corresponds to a
diff --git a/timeeval/metrics/thresholding.py b/timeeval/metrics/thresholding.py
index ab427488..2dfeb364 100644
--- a/timeeval/metrics/thresholding.py
+++ b/timeeval/metrics/thresholding.py
@@ -1,12 +1,12 @@
-import abc
 import contextlib
 import warnings
+from abc import ABC, abstractmethod
 from typing import Optional, Tuple, Any, Generator
 
 import numpy as np
 
 
-class ThresholdingStrategy(abc.ABC):
+class ThresholdingStrategy(ABC):
     """Takes an anomaly scoring and ground truth labels to compute and apply a threshold to the scoring.
 
     Subclasses of this abstract base class define different strategies to put a threshold over the anomaly scorings.
@@ -15,7 +15,7 @@ class ThresholdingStrategy(abc.ABC):
     already existing binary labels and keeps them untouched. This allows applying the metrics on existing binary
     classification results.
     """
-    def __int__(self) -> None:
+    def __init__(self) -> None:
         self.threshold: Optional[float] = None
 
     def fit(self, y_true: np.ndarray, y_score: np.ndarray) -> None:
@@ -70,10 +70,10 @@ def fit_transform(self, y_true: np.ndarray, y_score: np.ndarray) -> np.ndarray:
         self.fit(y_true, y_score)
         return self.transform(y_score)
 
-    @abc.abstractmethod
+    @abstractmethod
     def find_threshold(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
         """Abstract method containing the actual code to determine the threshold. Must be overwritten by subclasses!"""
-        pass
+        ...
 
 
 class NoThresholding(ThresholdingStrategy):
@@ -145,6 +145,7 @@ class FixedValueThresholding(ThresholdingStrategy):
         Fixed threshold to use. All anomaly scorings are scaled to the interval [0, 1]
     """
     def __init__(self, threshold: float = 0.8):
+        super().__init__()
         if threshold > 1 or threshold < 0:
             raise ValueError(f"Threshold must be in the interval [0, 1], but was {threshold}!")
         self.threshold = threshold
@@ -169,6 +170,7 @@ class PercentileThresholding(ThresholdingStrategy):
         The percentile of the anomaly scoring to use. Must be between 0 and 100.
     """
     def __init__(self, percentile: int = 90):
+        super().__init__()
         if percentile < 0 or percentile > 100:
             raise ValueError(f"Percentile must be within [0, 100], but was {percentile}!")
         self._percentile = percentile
@@ -207,6 +209,7 @@ class TopKPointsThresholding(ThresholdingStrategy):
         number of anomalous points.
     """
     def __init__(self, k: Optional[int] = None):
+        super().__init__()
         if k is not None and k <= 0:
             raise ValueError(f"K must be greater than 0, but was {k}!")
         self._k: Optional[int] = k
@@ -254,6 +257,7 @@ class TopKRangesThresholding(ThresholdingStrategy):
         anomalies.
     """
     def __init__(self, k: Optional[int] = None):
+        super().__init__()
         if k is not None and k <= 0:
             raise ValueError(f"K must be greater than 0, but was {k}!")
         self._k: Optional[int] = k
@@ -315,6 +319,7 @@ class SigmaThresholding(ThresholdingStrategy):
         Multiples of the standard deviation to be added to the mean to compute the threshold (:math:`x`).
     """
     def __init__(self, factor: float = 3.0):
+        super().__init__()
         if factor <= 0:
             raise ValueError(f"factor must be greater than 0, but was {factor}!")
         self._factor = factor
@@ -391,6 +396,7 @@ class PyThreshThresholding(ThresholdingStrategy):
     """
 
     def __init__(self, pythresh_thresholder: 'BaseThresholder', random_state: Any = None):  # type: ignore
+        super().__init__()
         self._thresholder = pythresh_thresholder
         self._predictions: Optional[np.ndarray] = None
         self._random_state: Any = random_state
diff --git a/timeeval/params/__init__.py b/timeeval/params/__init__.py
index d589471a..abce158c 100644
--- a/timeeval/params/__init__.py
+++ b/timeeval/params/__init__.py
@@ -1,1 +1,4 @@
-from .search import ParameterConfig, FullParameterGrid, IndependentParameterGrid, FixedParameters
+from .base import ParameterConfig, FixedParameters
+from .bayesian import BayesianParameterSearch
+from .grid_search import FullParameterGrid, IndependentParameterGrid
+from .params import Params
diff --git a/timeeval/params/base.py b/timeeval/params/base.py
new file mode 100644
index 00000000..615fc408
--- /dev/null
+++ b/timeeval/params/base.py
@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+import abc
+from typing import TYPE_CHECKING, Sized, Iterator, Mapping
+
+from .params import FixedParams
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+    from .params import Params
+    from typing import Any
+    from .params import Params
+
+
+class ParameterConfig(abc.ABC, Sized):
+    """Base class for algorithm hyperparameter configurations.
+
+    Currently, TimeEval supports three kinds of parameter configurations:
+
+    1. :class:`~timeeval.params.FixedParameters`: A single parameter setting with one value for each parameter.
+    2. :class:`~timeeval.params.IndependentParameterGrid` and :class:`~timeeval.params.FullParameterGrid`: Parameter
+       search using the specification of a parameter grid, where each parameter can have multiple values. Depending on
+       the parameter grid, TimeEval will build a parameter search space and test all combinations of parameters.
+    3. :class:`~timeeval.params.BayesianParameterSearch`: Parameter search using Bayesian optimization.
+
+    """
+
+    @abc.abstractmethod
+    def iter(self, algorithm: Algorithm, dataset: Dataset) -> Iterator[Params]:
+        """Iterate over the points in the grid.
+
+        Returns
+        -------
+        params : iterator over Params
+            Yields a params object that maps each parameter to a single value.
+        """
+        ...
+
+    @staticmethod
+    def defaults() -> ParameterConfig:
+        """Returns the default parameter configuration that has only a single parameter setting with no parameters."""
+        return FixedParameters({})
+
+
+class FixedParameters(ParameterConfig):
+    """Single parameters setting with one value for each.
+
+    Iterating over this grid yields the input setting as the first and only element.
+
+    Parameters
+    ----------
+    params : dict of str to Any
+        The parameter setting to be evaluated, as a dictionary mapping parameters to allowed values.
+        An empty dict signifies default parameters.
+
+    Examples
+    --------
+    >>> from timeeval.params import FixedParameters
+    >>> params = {"a": 2, "b": True}
+    >>> list(FixedParameters(params)) == (
+    ...    [{"a": 2, "b": True}])
+    True
+    >>> FixedParameters(params)[0] == {"a": 2, "b": True}
+    True
+    """
+
+    def __init__(self, params: Mapping[str, Any]):
+        if not isinstance(params, Mapping):
+            if isinstance(params, (list, Iterator)):
+                raise TypeError("A sequence of grids (Iterable[Mapping[str, Any]) is not supported by this "
+                                f"ParameterConfig ({params}). Please use a "
+                                "`timeeval.search.IndependentParameterGrid` for this!")
+            else:
+                raise TypeError(f"Parameters are not provided as a dict ({params})")
+        self._params = [FixedParams(params)]
+
+    def iter(self, algorithm: Algorithm, dataset: Dataset) -> Iterator[Params]:
+        return self.__iter__()
+
+    def __iter__(self) -> Iterator[Params]:
+        return iter(self._params)
+
+    def __len__(self) -> int:
+        return 1
+
+    def __getitem__(self, i: int) -> Params:
+        assert i == 0, "FixedParameters only has one element"
+        return self._params[i]
diff --git a/timeeval/params/bayesian.py b/timeeval/params/bayesian.py
new file mode 100644
index 00000000..a886cf1e
--- /dev/null
+++ b/timeeval/params/bayesian.py
@@ -0,0 +1,88 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from .base import ParameterConfig
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from optuna.distributions import BaseDistribution
+    from typing import Mapping, Iterator
+    from .params import Params
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
+    from ..integration.optuna import OptunaConfiguration, OptunaStudyConfiguration
+
+
+class BayesianParameterSearch(ParameterConfig):
+    """Performs Bayesian optimization using Optuna integration.
+
+    .. note::
+        Please install the `Optuna <https://optuna.org>`_ package to use this class.
+        If you use the recommended PostgreSQL storage backend, you also need to install the `psycopg2` or
+        `psycopg2-binary` package:
+
+        .. code-block:: bash
+
+            pip install optuna>=3.1.0 psycopg2
+
+    .. warning::
+        Parameter search using this class and the Optuna integration is **non-deterministic**. The results may vary
+        between different runs, even if the same seed is used (e.g., for the Optuna sampler or pruner). This is because
+        TimeEval needs to re-seed the Optuna samplers for every trial in distributed mode. This is necessary to ensure
+        that initial random samples are different over all workers.
+
+    Parameters
+    ----------
+    config : OptunaStudyConfiguration
+        Configuration for the Optuna study. Optional parameters are filled in with the default values from the global
+        Optuna configuration.
+    params : Mapping[str, BaseDistribution]
+        Mapping from parameter names to the corresponding Optuna distributions, such as
+        :class:`~optuna.distributions.IntDistribution`, :class:`~optuna.distributions.FloatDistribution`, or
+        :class:`~optuna.distributions.CategoricalDistribution`.
+
+    Examples
+    --------
+    >>> from timeeval.params import BayesianParameterSearch
+    >>> from timeeval.metrics import RangePrAUC
+    >>> from timeeval.integration.optuna import OptunaStudyConfiguration
+    >>> from optuna.distributions import FloatDistribution, IntDistribution
+    >>> config = OptunaStudyConfiguration(n_trials=10, metric=RangePrAUC())
+    >>> distributions = {
+    ...     "max_features": FloatDistribution(low=0.0, high=1.0, step=0.01),
+    ...     "window_size": IntDistribution(low=5, high=1000, step=5),
+    ... }
+    >>> BayesianParameterSearch(config, distributions)
+    <timeeval.params.bayesian.BayesianParameterSearch object at 0x7f9cdc8faf50>
+
+    See also
+    --------
+    `<https://optuna.readthedocs.io>`_:
+        Optuna documentation.
+    :class:`timeeval.integration.optuna.OptunaModule`:
+        Optuna integration TimeEval module.
+    """
+
+    def __init__(self, config: OptunaStudyConfiguration,
+                 params: Mapping[str, BaseDistribution],
+                 include_default_params: bool = False):
+        from ..integration.optuna import OptunaParameterSearch
+        self._impl = OptunaParameterSearch(config, params, include_default_params)
+
+    def iter(self, algorithm: Algorithm, dataset: Dataset) -> Iterator[Params]:
+        return self._impl.iter(algorithm, dataset)
+
+    def __len__(self) -> int:
+        return self._impl.__len__()
+
+    def update_config(self, global_config: OptunaConfiguration) -> None:
+        """Updates unset / default values in the study configuration with the global configuration values.
+
+        Parameters
+        ----------
+        global_config : OptunaConfiguration
+            Global Optuna configuration.
+        """
+        self._impl.update_config(global_config)
diff --git a/timeeval/params/search.py b/timeeval/params/grid_search.py
similarity index 65%
rename from timeeval/params/search.py
rename to timeeval/params/grid_search.py
index 63469edc..eac96f0b 100644
--- a/timeeval/params/search.py
+++ b/timeeval/params/grid_search.py
@@ -1,13 +1,24 @@
-import abc
-from collections.abc import Iterable, Sized
-from typing import Iterator, Any, Dict, Mapping, Optional
+from __future__ import annotations
+
+from abc import abstractmethod, ABCMeta
+from typing import TYPE_CHECKING, Mapping, Iterator, Iterable
 
 from sklearn.model_selection import ParameterGrid
 
+from .base import ParameterConfig
+from .params import FixedParams, Params
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    from typing import Any, Optional
+    from ..algorithm import Algorithm
+    from ..datasets import Dataset
 
-class ParameterConfig(abc.ABC, Iterable, Sized):
+
+class ParameterGridConfig(ParameterConfig, Iterable[Params], metaclass=ABCMeta):
     @property
-    @abc.abstractmethod
+    @abstractmethod
     def param_grid(self) -> ParameterGrid:
         """The parameter search grid.
 
@@ -16,31 +27,22 @@ def param_grid(self) -> ParameterGrid:
         param_grid: sklearn parameter grid object
             A parameter search grid compatible with sklearn: :class:`sklearn.model_selection.ParameterGrid`
         """
-        pass
+        ...
 
-    def __iter__(self) -> Iterator[Dict[str, Any]]:
-        """Iterate over the points in the grid.
+    def iter(self, algorithm: Algorithm, dataset: Dataset) -> Iterator[Params]:
+        return self.__iter__()
 
-        Returns
-        -------
-        params : iterator over dict of str to any
-            Yields dictionaries mapping each parameter to one of its allowed values.
-        """
-        return iter(self.param_grid)
+    def __iter__(self) -> Iterator[Params]:
+        return iter(FixedParams(p) for p in self.param_grid)
 
     def __len__(self) -> int:
-        """Number of points on the grid."""
         return len(self.param_grid)
 
-    def __getitem__(self, i: int) -> Dict[str, Any]:
-        return self.param_grid[i]  # type: ignore
+    def __getitem__(self, index: int) -> Params:
+        return FixedParams(self.param_grid[index])
 
-    @staticmethod
-    def defaults() -> "ParameterConfig":
-        return FixedParameters({})
 
-
-class FullParameterGrid(ParameterConfig):
+class FullParameterGrid(ParameterGridConfig):
     """Grid of parameters with a discrete number of values for each.
 
     Iterating over this grid yields the full cartesian product of all available parameter combinations. Uses the
@@ -54,7 +56,7 @@ class FullParameterGrid(ParameterConfig):
 
     Examples
     --------
-    >>> from timeeval.params.search import FullParameterGrid
+    >>> from timeeval.params import FullParameterGrid
     >>> params = {"a": [1, 2], "b": [True, False]}
     >>> list(FullParameterGrid(params)) == (
     ...    [{"a": 1, "b": True}, {"a": 1, "b": False},
@@ -84,7 +86,7 @@ def param_grid(self) -> ParameterGrid:
         return self._param_grid
 
 
-class IndependentParameterGrid(ParameterConfig):
+class IndependentParameterGrid(ParameterGridConfig):
     """Grid of parameters with a discrete number of values for each.
 
     The parameters in the dict are considered independent and explored one after the other (no cartesian product).
@@ -150,46 +152,3 @@ def __init__(self, param_grid: Mapping[str, Any], default_params: Optional[Mappi
     @property
     def param_grid(self) -> ParameterGrid:
         return self._param_grid
-
-
-class FixedParameters(ParameterConfig):
-    """Single parameters setting with one value for each.
-
-    Iterating over this grid yields the input setting as the first and only element. Uses the
-    sklearn.model_selection.ParameterGrid internally.
-
-    Parameters
-    ----------
-    params : dict of str to Any
-        The parameter setting to be evaluated, as a dictionary mapping parameters to allowed values.
-        An empty dict signifies default parameters.
-
-    Examples
-    --------
-    >>> from timeeval.params.search import FixedParameters
-    >>> params = {"a": 2, "b": True}
-    >>> list(FixedParameters(params)) == (
-    ...    [{"a": 2, "b": True}])
-    True
-    >>> FixedParameters(params)[0] == {"a": 2, "b": True}
-    True
-
-    See also
-    --------
-    :class:`sklearn.model_selection.ParameterGrid`:
-        Used internally to represent the parameter grids.
-    """
-
-    def __init__(self, params: Mapping[str, Any]):
-        if not isinstance(params, Mapping):
-            if isinstance(params, (list, Iterator)):
-                raise TypeError("A sequence of grids (Iterable[Mapping[str, Any]) is not supported by this "
-                                f"ParameterConfig ({params}). Please use a "
-                                "`timeeval.search.IndependentParameterGrid` for this!")
-            else:
-                raise TypeError(f"Parameters are not provided as a dict ({params})")
-        self._param_grid = ParameterGrid({k: [v] for k, v in params.items()})
-
-    @property
-    def param_grid(self) -> ParameterGrid:
-        return self._param_grid
diff --git a/timeeval/params/params.py b/timeeval/params/params.py
new file mode 100644
index 00000000..01394938
--- /dev/null
+++ b/timeeval/params/params.py
@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import TYPE_CHECKING, Any, Mapping, Dict
+
+from timeeval.utils.hash_dict import hash_dict
+
+
+# only imports the below classes for type checking to avoid circular imports (annotations-import is necessary!)
+if TYPE_CHECKING:
+    import numpy as np
+
+
+class Params(Mapping[str, Any], ABC):
+
+    @abstractmethod
+    def materialize(self) -> Params:
+        ...
+
+    @abstractmethod
+    def assess(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
+        ...
+
+    @abstractmethod
+    def fail(self) -> None:
+        ...
+
+    @abstractmethod
+    def uid(self) -> str:
+        ...
+
+    @abstractmethod
+    def to_dict(self) -> Dict[str, Any]:
+        ...
+
+
+class FixedParams(Dict[str, Any], Params):
+    def materialize(self) -> FixedParams:
+        return self
+
+    def assess(self, y_true: np.ndarray, y_score: np.ndarray) -> float:
+        return 0.0
+
+    def fail(self) -> None:
+        pass
+
+    def uid(self) -> str:
+        return hash_dict(self)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return self
diff --git a/timeeval/remote_configuration.py b/timeeval/remote_configuration.py
index 94b4fec5..922f24e6 100644
--- a/timeeval/remote_configuration.py
+++ b/timeeval/remote_configuration.py
@@ -59,7 +59,7 @@ class RemoteConfiguration:
     scheduler_port: int = DEFAULT_DASK_PORT
     worker_hosts: List[str] = field(default_factory=lambda: [])
     remote_python: str = field(default_factory=lambda: sys.executable)
-    kwargs_overwrites: dict = field(default_factory=lambda: {})
+    kwargs_overwrites: Dict[str, Any] = field(default_factory=lambda: {})
     dask_logging_file_level: str = "INFO"
     dask_logging_console_level: str = "INFO"
     dask_logging_filename: str = DEFAULT_DASK_LOG_FILENAME
diff --git a/timeeval/resource_constraints.py b/timeeval/resource_constraints.py
index c2b18bc8..83f48f94 100644
--- a/timeeval/resource_constraints.py
+++ b/timeeval/resource_constraints.py
@@ -1,9 +1,12 @@
+from __future__ import annotations
+
 from dataclasses import dataclass
 from typing import Optional, Tuple
 
 import psutil
 from durations import Duration
 
+
 MB = 1024 ** 2
 """:math:`1 MB = 2^{20} \\text{Bytes}`
 
@@ -175,6 +178,6 @@ def _get_timeout_with_overwrite(timeout: Duration, overwrite: Optional[Duration]
         return timeout
 
     @staticmethod
-    def default_constraints() -> 'ResourceConstraints':
+    def default_constraints() -> ResourceConstraints:
         """Creates a configuration object with the default resource constraints."""
         return ResourceConstraints()
diff --git a/timeeval/timeeval.py b/timeeval/timeeval.py
index 689a7e3c..e3e72ecb 100644
--- a/timeeval/timeeval.py
+++ b/timeeval/timeeval.py
@@ -7,7 +7,7 @@
 from pathlib import Path
 from time import time
 from types import FrameType
-from typing import Callable, List, Tuple, Dict, Optional, Any
+from typing import Callable, List, Tuple, Dict, Optional, Any, Union, Mapping
 
 import numpy as np
 import pandas as pd
@@ -15,17 +15,19 @@
 from distributed.client import Future
 from joblib import Parallel, delayed
 
+from ._core.experiments import Experiments, Experiment
+from ._core.remote import Remote, RemoteConfiguration
+from ._core.times import Times
 from .adapters.docker import DockerTimeoutError, DockerMemoryError, DockerAdapter
 from .algorithm import Algorithm
 from .constants import RESULTS_CSV
-from .core.experiments import Experiments, Experiment
-from .core.remote import Remote, RemoteConfiguration
-from .core.times import Times
 from .data_types import TrainingType, InputDimensionality
 from .datasets import Datasets
+from .integration import TimeEvalModule
+from .metrics import Metric, DefaultMetrics
+from .params import BayesianParameterSearch
 from .resource_constraints import ResourceConstraints
 from .utils.encode_params import dumps_params
-from .metrics import Metric, DefaultMetrics
 from .utils.tqdm_joblib import tqdm_joblib
 
 
@@ -140,13 +142,13 @@ class TimeEval:
         - univariate algorithms on multivariate datasets (algorithm cannot process the dataset)
 
     force_training_type_match : bool
-        Narrow down the algorithm-dataset combinations further by executing an algorithm only on datasets with **the same**
-        training type, e.g. unsupervised algorithms only on unsupervised datasets.
-        This flag implies ``skip_invalid_combinations==True``.
+        Narrow down the algorithm-dataset combinations further by executing an algorithm only on datasets with
+        **the same** training type, e.g. unsupervised algorithms only on unsupervised datasets. This flag implies
+        ``skip_invalid_combinations==True``.
     force_dimensionality_match : bool
-        Narrow down the algorithm-dataset combinations furthter by executing an algorithm only on datasets with **the same**
-        input dimensionality, e.g. multivariate algorithms only on multivariate datasets.
-        This flag implies ``skip_invalid_combinations==True``.
+        Narrow down the algorithm-dataset combinations furthter by executing an algorithm only on datasets with
+        **the same** input dimensionality, e.g. multivariate algorithms only on multivariate datasets. This flag implies
+        ``skip_invalid_combinations==True``.
     n_jobs : int
         Set the number of jobs / processes used to fetch the results from the remote machine.
         This setting is used only in distributed mode.
@@ -164,6 +166,14 @@ class TimeEval:
 
         Only experiments that are present in the TimeEval configuration **and** this file are scheduled and executed.
         This allows you to circumvent the cross-product that TimeEval will perform in its default configuration.
+    module_configs : Mapping[str, Any], optional
+        Use this parameter to pass additional configuration options for automatically loaded TimeEval modules. This is
+        currently used only for the implementation of the Bayesian hyperparameter optimization prozedure using Optuna.
+        See :class:`timeeval.integration.optuna.OptunaModule` and :class:`timeeval.params.bayesian.OptunaParameterSearch`
+        for details.
+
+        You can access loaded modules via the ``modules`` attribute (``Dict[str, TimeEvalModule``) of the TimeEval
+        instance, e.g. ``timeeval.modules["optuna"]``.
     """
 
     RESULT_KEYS = ["algorithm",
@@ -218,7 +228,8 @@ def __init__(self,
                  force_training_type_match: bool = False,
                  force_dimensionality_match: bool = False,
                  n_jobs: int = -1,
-                 experiment_combinations_file: Optional[Path] = None) -> None:
+                 experiment_combinations_file: Optional[Path] = None,
+                 module_configs: Mapping[str, Any] = {}) -> None:
         assert len(datasets) > 0, "No datasets given for evaluation!"
         assert len(algorithms) > 0, "No algorithms given for evaluation!"
         assert repetitions > 0, "Negative or 0 repetitions are not supported!"
@@ -265,6 +276,20 @@ def __init__(self,
             )
             limits.tasks_per_host = 1
 
+        # load necessary modules:
+        self.modules: Dict[str, TimeEvalModule] = {}
+        # Optuna
+        if any(isinstance(a.param_config, BayesianParameterSearch) for a in algorithms):
+            if repetitions > 1:
+                self.log.warning(
+                    f"`repetitions` was set to {repetitions}. However, some algorithms require Optuna for parameter "
+                    "search, which does not support repetitions. Reducing `repetitions` for all experiments to 1."
+                )
+                repetitions = 1
+            from .integration.optuna import OptunaModule, OptunaConfiguration
+            optuna_config = module_configs.get("optuna", OptunaConfiguration.default(self.distributed))
+            self.modules["optuna"] = OptunaModule(optuna_config)
+
         self.exps = Experiments(dataset_mgr, dataset_details, algorithms, self.results_path,
                                 resource_constraints=limits,
                                 repetitions=repetitions,
@@ -302,7 +327,7 @@ def _run(self) -> None:
         for exp in tqdm.tqdm(self.exps, desc=desc, disable=self.disable_progress_bar):
             try:
                 future_result: Optional[Future] = None
-                result: Optional[Dict] = None
+                result: Optional[Dict[str, Any]] = None
 
                 if exp.algorithm.training_type in [TrainingType.SUPERVISED, TrainingType.SEMI_SUPERVISED]:
                     if exp.resolved_train_dataset_path is None:
@@ -342,7 +367,7 @@ def _run(self) -> None:
 
     def _record_results(self,
                         exp: Experiment,
-                        result: Optional[Dict] = None,
+                        result: Optional[Dict[str, Any]] = None,
                         future_result: Optional[Future] = None,
                         status: Status = Status.OK,
                         error_message: Optional[str] = None) -> None:
@@ -357,20 +382,23 @@ def _record_results(self,
             "status": status,
             "error_message": error_message,
             "repetition": exp.repetition,
-            "hyper_params": dumps_params(exp.params),
             "hyper_params_id": exp.params_id
         }
+        try:
+            new_row["hyper_params"] = dumps_params(exp.params)
+        except ValueError:
+            pass
         if result is not None and future_result is None:
             new_row.update(result)
         elif result is None and future_result is not None:
             new_row.update({"future_result": future_result})
-        self.results = self.results.append(new_row, ignore_index=True)
+        self.results = pd.concat([self.results, pd.DataFrame([new_row])], ignore_index=True)
         self.results.replace(to_replace=[None], value=np.nan, inplace=True)
 
     def _resolve_future_results(self) -> None:
         self.remote.fetch_results()
 
-        result_keys = self.metric_names + Times.result_keys()
+        result_keys = ["hyper_params"] + self.metric_names + Times.result_keys()
         status_keys = ["status", "error_message"]
         keys = result_keys + status_keys
 
@@ -531,7 +559,7 @@ def _finalize(self) -> None:
             algorithm.finalize()
 
     def _distributed_prepare(self) -> None:
-        tasks: List[Tuple[Callable, List, Dict]] = []
+        tasks: List[Tuple[Union[Callable[[List[Path]], None], Callable[[], None]], List[Any], Dict[str, Any]]] = []
         for algorithm in self.exps.algorithms:
             prepare_fn = algorithm.prepare_fn()
             if prepare_fn:
@@ -548,7 +576,7 @@ def mkdirs(dirs: List[Path]) -> None:
         self.remote.run_on_all_hosts(tasks, msg="Preparing")
 
     def _distributed_finalize(self) -> None:
-        tasks: List[Tuple[Callable, List, Dict]] = []
+        tasks: List[Tuple[Callable[[], None], List[Any], Dict[str, Any]]] = []
         for algorithm in self.exps.algorithms:
             finalize_fn = algorithm.finalize_fn()
             if finalize_fn:
@@ -556,10 +584,6 @@ def _distributed_finalize(self) -> None:
         self.log.debug(f"Collected {len(tasks)} algorithm finalize steps")
         self.log.info("Running finalize steps on remote hosts")
         self.remote.run_on_all_hosts(tasks, msg="Finalizing")
-        self.log.info("Closing remote")
-        self.remote.close()
-        self.log.info("Syncing results")
-        self.rsync_results()
 
     def run(self) -> None:
         """Starts the configured evaluation run.
@@ -583,6 +607,8 @@ def run(self) -> None:
         print("Running PREPARE phase")
         self.log.info("Running PREPARE phase")
         t0 = time()
+        for module in self.modules.values():
+            module.prepare(self)
         if self.distributed:
             self._distributed_prepare()
         else:
@@ -590,9 +616,13 @@ def run(self) -> None:
 
         print("Running EVALUATION phase")
         self.log.info("Running EVALUATION phase")
+        for module in self.modules.values():
+            module.pre_run(self)
         self._run()
         if self.distributed:
             self._resolve_future_results()
+        for module in self.modules.values():
+            module.post_run(self)
         self.save_results()
 
         print("Running FINALIZE phase")
@@ -601,6 +631,14 @@ def run(self) -> None:
             self._distributed_finalize()
         else:
             self._finalize()
+        for module in self.modules.values():
+            module.finalize(self)
+
+        if self.distributed:
+            self.log.info("Closing remote")
+            self.remote.close()
+            self.log.info("Syncing results")
+            self.rsync_results()
         t1 = time()
 
         msg = f"""FINALIZE phase done.
diff --git a/timeeval/utils/encode_params.py b/timeeval/utils/encode_params.py
index ae8500e4..d75bc338 100755
--- a/timeeval/utils/encode_params.py
+++ b/timeeval/utils/encode_params.py
@@ -1,18 +1,20 @@
 import json
 from pathlib import Path
-from typing import Dict, Any, TextIO, Union
+from typing import TextIO, Union
 
 from numpyencoder import NumpyEncoder
 
+from timeeval.params import Params
 
-def dumps_params(params: Dict[str, Any]) -> str:
-    return json.dumps(params, cls=NumpyEncoder)
 
+def dumps_params(params: Params) -> str:
+    return json.dumps(params.to_dict(), cls=NumpyEncoder)
 
-def dump_params(params: Dict[str, Any], fh: Union[str, Path, TextIO]) -> None:
+
+def dump_params(params: Params, fh: Union[str, Path, TextIO]) -> None:
     if isinstance(fh, str):
         fh = Path(fh)
     if isinstance(fh, Path):
         fh = fh.open("w", encoding="utf-8")
-    json.dump(params, fh, cls=NumpyEncoder)
+    json.dump(params.to_dict(), fh, cls=NumpyEncoder)
     fh.close()
diff --git a/timeeval/utils/hash_dict.py b/timeeval/utils/hash_dict.py
index adb3b2aa..465463e9 100644
--- a/timeeval/utils/hash_dict.py
+++ b/timeeval/utils/hash_dict.py
@@ -1,4 +1,5 @@
 from hashlib import md5
+from typing import Any, Mapping
 
 
-def hash_dict(x: dict) -> str: return md5(str(sorted(x.items())).encode("utf-8")).hexdigest()
+def hash_dict(x: Mapping[Any, Any]) -> str: return md5(str(sorted(x.items())).encode("utf-8")).hexdigest()
diff --git a/timeeval_experiments/2023-01-30-BO-baselines.py b/timeeval_experiments/2023-01-30-BO-baselines.py
new file mode 100644
index 00000000..549ace4d
--- /dev/null
+++ b/timeeval_experiments/2023-01-30-BO-baselines.py
@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+import logging
+import sys
+import random
+from typing import List, Tuple
+
+import numpy as np
+from durations import Duration
+from optuna import distributions
+
+from timeeval import TimeEval, MultiDatasetManager, TrainingType, InputDimensionality, RemoteConfiguration
+from timeeval.constants import HPI_CLUSTER
+from timeeval.integration.optuna import OptunaConfiguration, OptunaStudyConfiguration
+from timeeval.metrics import RangePrAUC, RangeRocAUC
+from timeeval.params import BayesianParameterSearch
+from timeeval.resource_constraints import ResourceConstraints, GB
+from timeeval_experiments.algorithms import subsequence_knn, subsequence_lof, subsequence_if, stomp, kmeans, \
+    dwt_mlead, torsk
+
+
+# Setup logging
+logging.basicConfig(
+    filename="timeeval.log",
+    filemode="a",
+    level=logging.INFO,
+    # force=True,
+    format="%(asctime)s %(levelname)6.6s - %(name)20.20s: %(message)s",
+)
+
+random.seed(42)
+np.random.rand(42)
+
+def main():
+    dm = MultiDatasetManager([
+        HPI_CLUSTER.akita_dataset_paths[HPI_CLUSTER.BENCHMARK],
+        HPI_CLUSTER.akita_dataset_paths[HPI_CLUSTER.UNIVARIATE_ANOMALY_TEST_CASES],
+    ])
+
+    # Select datasets and algorithms
+    datasets: List[Tuple[str, str]] = dm.select(collection="Dodgers")
+    datasets += random.sample(dm.select(
+        input_dimensionality=InputDimensionality.UNIVARIATE,
+        training_type=TrainingType.UNSUPERVISED,
+        collection="univariate-anomaly-test-cases",
+    ), 3)
+    datasets += dm.select(dataset="sine-difflen-3-frequency")
+    datasets += dm.select(dataset="sine-difflen-3-variance")
+    datasets += random.sample(dm.select(collection="KDD-TSAD", max_contamination=0.1), 2)
+    datasets += random.sample(dm.select(collection="NASA-MSL", max_contamination=0.1), 2)
+
+    study_config = OptunaStudyConfiguration(n_trials=300, metric=RangePrAUC(buffer_size=100))
+    algorithms = [
+        dwt_mlead(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "quantile_epsilon": distributions.FloatDistribution(0.001, 0.5),
+            "start_level": distributions.IntDistribution(1, 10),
+        })),
+        kmeans(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "anomaly_window_size": distributions.IntDistribution(10, 2000),
+            "n_clusters": distributions.IntDistribution(2, 100),
+        })),
+        stomp(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "anomaly_window_size": distributions.IntDistribution(10, 2000),
+            "exclusion_zone": distributions.FloatDistribution(0, 0.75),
+        })),
+        subsequence_if(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "window_size": distributions.IntDistribution(10, 2000),
+            "n_trees": distributions.IntDistribution(10, 1000),
+            "bootstrap": distributions.CategoricalDistribution([True, False]),
+            "max_features": distributions.FloatDistribution(0.01, 1.0),
+            "max_samples": distributions.FloatDistribution(0.01, 1.0),
+        })),
+        subsequence_lof(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "window_size": distributions.IntDistribution(10, 2000),
+            "distance_metric_order": distributions.IntDistribution(1, 4),
+            "leaf_size": distributions.IntDistribution(10, 100),
+            "n_neighbors": distributions.IntDistribution(1, 100),
+        })),
+        subsequence_knn(BayesianParameterSearch(study_config, include_default_params=True, params={
+            "window_size": distributions.IntDistribution(10, 2000),
+            "distance_metric_order": distributions.IntDistribution(1, 4),
+            "leaf_size": distributions.IntDistribution(10, 100),
+            "n_neighbors": distributions.IntDistribution(1, 100),
+            "method": distributions.CategoricalDistribution(["largest", "mean", "median"]),
+            "radius": distributions.FloatDistribution(0.1, 5.0),
+        })),
+        torsk(BayesianParameterSearch(study_config.copy(n_trials=900), include_default_params=True, params={
+            "context_window_size": distributions.IntDistribution(10, 2000),
+            "density": distributions.FloatDistribution(0.0001, 1, log=True),
+            "imed_loss": distributions.CategoricalDistribution([True, False]),
+            "input_map_scale": distributions.FloatDistribution(0.001, 1.0),
+            "input_map_size": distributions.IntDistribution(10, 2000),
+            "prediction_window_size": distributions.IntDistribution(10, 1000),
+            "scoring_large_window_size": distributions.IntDistribution(50, 2000),
+            "scoring_small_window_size": distributions.IntDistribution(10, 1000),
+            "spectral_radius": distributions.FloatDistribution(1.0, 10.0),
+            "train_window_size": distributions.IntDistribution(10, 1000),
+            "transient_window_size": distributions.IntDistribution(10, 1000),
+        })),
+    ]
+
+    print("\nDatasets:")
+    print("=====================================================================================")
+    for collection in np.unique([c for (c, d) in datasets]):
+        print(collection)
+        cds = sorted([d for (c, d) in datasets if c == collection])
+        for cd in cds:
+            print(f"  {cd}")
+    print("=====================================================================================\n\n")
+
+    print("\nParameter configurations:")
+    print("=====================================================================================")
+    for algo in algorithms:
+        print(f"{algo.name}: {len(algo.param_config)}")
+    print("=====================================================================================\n\n")
+    print(f"Datasets: {len(datasets)}")
+    print(f"Algorithms: {len(algorithms)}")
+    sys.stdout.flush()
+
+    optuna_config = OptunaConfiguration(
+        default_storage="postgresql",
+        dashboard=True,
+        remove_managed_containers=False
+    )
+    cluster_config = RemoteConfiguration(
+        scheduler_host=HPI_CLUSTER.odin01,
+        worker_hosts=list(set(HPI_CLUSTER.nodes) - {HPI_CLUSTER.odin02, HPI_CLUSTER.odin03}),
+    )
+    limits = ResourceConstraints(
+        tasks_per_host=10,
+        task_cpu_limit=1.,
+        task_memory_limit=6 * GB,
+        execute_timeout=Duration("4 hours"),
+    )
+    timeeval = TimeEval(dm, datasets, algorithms,
+                        repetitions=1,
+                        distributed=True,
+                        remote_config=cluster_config,
+                        resource_constraints=limits,
+                        skip_invalid_combinations=True,
+                        metrics=[RangeRocAUC(buffer_size=100), RangePrAUC(buffer_size=100)],
+                        module_configs={"optuna": optuna_config},
+                        )
+
+    timeeval.run()
+    print(timeeval.get_results(aggregated=True, short=True))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/timeeval_experiments/algorithm_configurator.py b/timeeval_experiments/algorithm_configurator.py
index 717e615c..4af9b966 100755
--- a/timeeval_experiments/algorithm_configurator.py
+++ b/timeeval_experiments/algorithm_configurator.py
@@ -26,7 +26,7 @@ def __init__(self, config_path: Union[str, Path], check: bool = True):
         if check and self._heuristic_mapping:
             self._check_heuristics()
 
-    def _check_heuristics(self):
+    def _check_heuristics(self) -> None:
         broken_heuristics = []
         for key in self._heuristic_mapping:
             signature = self._heuristic_mapping[key]
@@ -41,8 +41,8 @@ def _check_heuristics(self):
         else:
             print("Heuristics are valid.")
 
-    def _substitute_heuristics(self, search_space: List[Any], checked: bool = False) -> Any:
-        def substitute(v):
+    def _substitute_heuristics(self, search_space: List[Any], checked: bool = False) -> List[Any]:
+        def substitute(v: Any) -> Any:
             try:
                 return f"heuristic:{self._heuristic_mapping[v]}"
             except (KeyError, TypeError) as e:
diff --git a/timeeval_experiments/algorithms/arima.py b/timeeval_experiments/algorithms/arima.py
index d0188416..9a102ee9 100644
--- a/timeeval_experiments/algorithms/arima.py
+++ b/timeeval_experiments/algorithms/arima.py
@@ -64,7 +64,7 @@
 }
 
 
-def arima(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def arima(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="ARIMA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/autoencoder.py b/timeeval_experiments/algorithms/autoencoder.py
index f1793ab1..b4ea9206 100644
--- a/timeeval_experiments/algorithms/autoencoder.py
+++ b/timeeval_experiments/algorithms/autoencoder.py
@@ -52,7 +52,7 @@
 }
 
 
-def autoencoder(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def autoencoder(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="AutoEncoder (AE)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/bagel.py b/timeeval_experiments/algorithms/bagel.py
index 2a52f853..83aa144e 100644
--- a/timeeval_experiments/algorithms/bagel.py
+++ b/timeeval_experiments/algorithms/bagel.py
@@ -79,7 +79,7 @@
 }
 
 
-def bagel(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def bagel(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Bagel",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/baseline_increasing.py b/timeeval_experiments/algorithms/baseline_increasing.py
index 35185355..f27408de 100644
--- a/timeeval_experiments/algorithms/baseline_increasing.py
+++ b/timeeval_experiments/algorithms/baseline_increasing.py
@@ -9,7 +9,7 @@
 _baseline_increasing_parameters: Dict[str, Dict[str, Any]] = {}
 
 
-def baseline_increasing(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def baseline_increasing(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Increasing Baseline",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/baseline_normal.py b/timeeval_experiments/algorithms/baseline_normal.py
index d8633749..6f42de70 100644
--- a/timeeval_experiments/algorithms/baseline_normal.py
+++ b/timeeval_experiments/algorithms/baseline_normal.py
@@ -9,7 +9,7 @@
 _baseline_normal_parameters: Dict[str, Dict[str, Any]] = {}
 
 
-def baseline_normal(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def baseline_normal(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Normal Baseline",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/baseline_random.py b/timeeval_experiments/algorithms/baseline_random.py
index cbf4a818..6a27bf0d 100644
--- a/timeeval_experiments/algorithms/baseline_random.py
+++ b/timeeval_experiments/algorithms/baseline_random.py
@@ -16,7 +16,7 @@
 }
 
 
-def baseline_random(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def baseline_random(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Random Baseline",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/cblof.py b/timeeval_experiments/algorithms/cblof.py
index 6e760b09..506a35e3 100644
--- a/timeeval_experiments/algorithms/cblof.py
+++ b/timeeval_experiments/algorithms/cblof.py
@@ -46,7 +46,7 @@
 }
 
 
-def cblof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def cblof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="CBLOF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/cof.py b/timeeval_experiments/algorithms/cof.py
index 663db3d1..0ce219f8 100644
--- a/timeeval_experiments/algorithms/cof.py
+++ b/timeeval_experiments/algorithms/cof.py
@@ -22,7 +22,7 @@
 }
 
 
-def cof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def cof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="COF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/copod.py b/timeeval_experiments/algorithms/copod.py
index 414df827..04c0b916 100644
--- a/timeeval_experiments/algorithms/copod.py
+++ b/timeeval_experiments/algorithms/copod.py
@@ -16,7 +16,7 @@
 }
 
 
-def copod(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def copod(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="COPOD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/dae.py b/timeeval_experiments/algorithms/dae.py
index 33190944..929efd3f 100644
--- a/timeeval_experiments/algorithms/dae.py
+++ b/timeeval_experiments/algorithms/dae.py
@@ -58,7 +58,7 @@
 }
 
 
-def dae(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def dae(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DenoisingAutoEncoder (DAE)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/damp.py b/timeeval_experiments/algorithms/damp.py
index 60a1e9bc..1858f978 100644
--- a/timeeval_experiments/algorithms/damp.py
+++ b/timeeval_experiments/algorithms/damp.py
@@ -49,7 +49,7 @@ def post_damp(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def damp(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def damp(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DAMP",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/dbstream.py b/timeeval_experiments/algorithms/dbstream.py
index e9f8f27e..fbe7dfa4 100644
--- a/timeeval_experiments/algorithms/dbstream.py
+++ b/timeeval_experiments/algorithms/dbstream.py
@@ -64,7 +64,7 @@
 }
 
 
-def dbstream(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def dbstream(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DBStream",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/deepant.py b/timeeval_experiments/algorithms/deepant.py
index 8a0321f3..17b3950c 100644
--- a/timeeval_experiments/algorithms/deepant.py
+++ b/timeeval_experiments/algorithms/deepant.py
@@ -75,7 +75,7 @@ def _post_deepant(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def deepant(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def deepant(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DeepAnT",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/deepnap.py b/timeeval_experiments/algorithms/deepnap.py
index 56139839..61104fb5 100644
--- a/timeeval_experiments/algorithms/deepnap.py
+++ b/timeeval_experiments/algorithms/deepnap.py
@@ -103,7 +103,7 @@ def post_deepnap(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def deepnap(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def deepnap(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DeepNAP",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/donut.py b/timeeval_experiments/algorithms/donut.py
index b882088f..a30a5ce6 100644
--- a/timeeval_experiments/algorithms/donut.py
+++ b/timeeval_experiments/algorithms/donut.py
@@ -61,7 +61,7 @@ def post_donut(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def donut(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def donut(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Donut",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/dspot.py b/timeeval_experiments/algorithms/dspot.py
index 6d8c7f74..bc3f1bb6 100644
--- a/timeeval_experiments/algorithms/dspot.py
+++ b/timeeval_experiments/algorithms/dspot.py
@@ -64,7 +64,7 @@
 }
 
 
-def dspot(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def dspot(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DSPOT",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/dwt_mlead.py b/timeeval_experiments/algorithms/dwt_mlead.py
index 89b27377..bfd83123 100644
--- a/timeeval_experiments/algorithms/dwt_mlead.py
+++ b/timeeval_experiments/algorithms/dwt_mlead.py
@@ -34,7 +34,7 @@
 }
 
 
-def dwt_mlead(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def dwt_mlead(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="DWT-MLEAD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/eif.py b/timeeval_experiments/algorithms/eif.py
index 4a612749..7a0af8e7 100644
--- a/timeeval_experiments/algorithms/eif.py
+++ b/timeeval_experiments/algorithms/eif.py
@@ -40,7 +40,7 @@
 }
 
 
-def eif(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def eif(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Extended Isolation Forest (EIF)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/encdec_ad.py b/timeeval_experiments/algorithms/encdec_ad.py
index 45451cf9..753fe859 100644
--- a/timeeval_experiments/algorithms/encdec_ad.py
+++ b/timeeval_experiments/algorithms/encdec_ad.py
@@ -97,7 +97,7 @@ def post_encdec_ad(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def encdec_ad(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def encdec_ad(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="EncDec-AD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/ensemble_gi.py b/timeeval_experiments/algorithms/ensemble_gi.py
index 6ee91e1a..5cfea455 100644
--- a/timeeval_experiments/algorithms/ensemble_gi.py
+++ b/timeeval_experiments/algorithms/ensemble_gi.py
@@ -58,7 +58,7 @@
 }
 
 
-def ensemble_gi(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def ensemble_gi(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Ensemble GI",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/fast_mcd.py b/timeeval_experiments/algorithms/fast_mcd.py
index 0d658afe..36988415 100644
--- a/timeeval_experiments/algorithms/fast_mcd.py
+++ b/timeeval_experiments/algorithms/fast_mcd.py
@@ -28,7 +28,7 @@
 }
 
 
-def fast_mcd(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def fast_mcd(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Fast-MCD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/fft.py b/timeeval_experiments/algorithms/fft.py
index 58c4e3ae..61e317d9 100644
--- a/timeeval_experiments/algorithms/fft.py
+++ b/timeeval_experiments/algorithms/fft.py
@@ -46,7 +46,7 @@
 }
 
 
-def fft(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def fft(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="FFT",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/generic_rf.py b/timeeval_experiments/algorithms/generic_rf.py
index c9a2d548..c04fb8d7 100644
--- a/timeeval_experiments/algorithms/generic_rf.py
+++ b/timeeval_experiments/algorithms/generic_rf.py
@@ -76,7 +76,7 @@
 }
 
 
-def generic_rf(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def generic_rf(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Random Forest Regressor (RR)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/generic_xgb.py b/timeeval_experiments/algorithms/generic_xgb.py
index 838c4981..3fc15256 100644
--- a/timeeval_experiments/algorithms/generic_xgb.py
+++ b/timeeval_experiments/algorithms/generic_xgb.py
@@ -94,7 +94,7 @@
 }
 
 
-def generic_xgb(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def generic_xgb(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="XGBoosting (RR)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/grammarviz3.py b/timeeval_experiments/algorithms/grammarviz3.py
index 1608e414..e54674ca 100644
--- a/timeeval_experiments/algorithms/grammarviz3.py
+++ b/timeeval_experiments/algorithms/grammarviz3.py
@@ -40,7 +40,7 @@ def post_grammarviz(algorithm_parameter: AlgorithmParameter, args: dict) -> np.n
     scores = np.zeros_like(sums)
     np.divide(sums, counts, out=scores, where=counts != 0)
     # returns the completely flattened array (from `[[1.2], [2.3]]` to `[1.2, 2.3]`)
-    return scores.A1
+    return scores.A1  # type: ignore
 
 
 _grammarviz3_parameters: Dict[str, Dict[str, Any]] = {
@@ -83,7 +83,7 @@ def post_grammarviz(algorithm_parameter: AlgorithmParameter, args: dict) -> np.n
 }
 
 
-def grammarviz3(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def grammarviz3(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="GrammarViz",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/grammarviz3_multi.py b/timeeval_experiments/algorithms/grammarviz3_multi.py
index 13969501..59aeff57 100644
--- a/timeeval_experiments/algorithms/grammarviz3_multi.py
+++ b/timeeval_experiments/algorithms/grammarviz3_multi.py
@@ -64,7 +64,7 @@
 }
 
 
-def grammarviz3_multi(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def grammarviz3_multi(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="GrammarViz-Multivariate",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/hbos.py b/timeeval_experiments/algorithms/hbos.py
index 81a5c042..20121f67 100644
--- a/timeeval_experiments/algorithms/hbos.py
+++ b/timeeval_experiments/algorithms/hbos.py
@@ -34,7 +34,7 @@
 }
 
 
-def hbos(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def hbos(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="HBOS",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/health_esn.py b/timeeval_experiments/algorithms/health_esn.py
index 43f01637..6b8b9f6d 100644
--- a/timeeval_experiments/algorithms/health_esn.py
+++ b/timeeval_experiments/algorithms/health_esn.py
@@ -46,7 +46,7 @@
 }
 
 
-def health_esn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def health_esn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="HealthESN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/hif.py b/timeeval_experiments/algorithms/hif.py
index 645e1944..b62b5f99 100644
--- a/timeeval_experiments/algorithms/hif.py
+++ b/timeeval_experiments/algorithms/hif.py
@@ -28,7 +28,7 @@
 }
 
 
-def hif(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def hif(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Hybrid Isolation Forest (HIF)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/hotsax.py b/timeeval_experiments/algorithms/hotsax.py
index 5c875436..17e5ec76 100644
--- a/timeeval_experiments/algorithms/hotsax.py
+++ b/timeeval_experiments/algorithms/hotsax.py
@@ -41,7 +41,7 @@ def post_hotsax(algorithm_parameter: AlgorithmParameter, args: dict) -> np.ndarr
     scores = np.zeros_like(sums)
     np.divide(sums, counts, out=scores, where=counts != 0)
     # returns the completely flattened array (from `[[1.2], [2.3]]` to `[1.2, 2.3]`)
-    return scores.A1
+    return scores.A1  # type: ignore
 
 
 _hotsax_parameters: Dict[str, Dict[str, Any]] = {
@@ -84,7 +84,7 @@ def post_hotsax(algorithm_parameter: AlgorithmParameter, args: dict) -> np.ndarr
 }
 
 
-def hotsax(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def hotsax(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="HOT SAX",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/hybrid_knn.py b/timeeval_experiments/algorithms/hybrid_knn.py
index 054e1787..02015bec 100644
--- a/timeeval_experiments/algorithms/hybrid_knn.py
+++ b/timeeval_experiments/algorithms/hybrid_knn.py
@@ -94,7 +94,7 @@ def post_hybrid_knn(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def hybrid_knn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def hybrid_knn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Hybrid KNN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/if_lof.py b/timeeval_experiments/algorithms/if_lof.py
index 833ee6fa..9a792c23 100644
--- a/timeeval_experiments/algorithms/if_lof.py
+++ b/timeeval_experiments/algorithms/if_lof.py
@@ -46,7 +46,7 @@
 }
 
 
-def if_lof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def if_lof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="IF-LOF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/iforest.py b/timeeval_experiments/algorithms/iforest.py
index 3cba0ff0..b18a89d8 100644
--- a/timeeval_experiments/algorithms/iforest.py
+++ b/timeeval_experiments/algorithms/iforest.py
@@ -52,7 +52,7 @@
 }
 
 
-def iforest(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def iforest(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Isolation Forest (iForest)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/img_embedding_cae.py b/timeeval_experiments/algorithms/img_embedding_cae.py
index c8fdf12b..81cba97e 100644
--- a/timeeval_experiments/algorithms/img_embedding_cae.py
+++ b/timeeval_experiments/algorithms/img_embedding_cae.py
@@ -88,7 +88,7 @@
 }
 
 
-def img_embedding_cae(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def img_embedding_cae(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="ImageEmbeddingCAE",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/kmeans.py b/timeeval_experiments/algorithms/kmeans.py
index 50b1610b..ee403ee3 100644
--- a/timeeval_experiments/algorithms/kmeans.py
+++ b/timeeval_experiments/algorithms/kmeans.py
@@ -40,7 +40,7 @@
 }
 
 
-def kmeans(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def kmeans(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="k-Means",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/knn.py b/timeeval_experiments/algorithms/knn.py
index be533f17..93e0b497 100644
--- a/timeeval_experiments/algorithms/knn.py
+++ b/timeeval_experiments/algorithms/knn.py
@@ -52,7 +52,7 @@
 }
 
 
-def knn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def knn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="KNN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/laser_dbn.py b/timeeval_experiments/algorithms/laser_dbn.py
index 8d47a1b3..340fe595 100644
--- a/timeeval_experiments/algorithms/laser_dbn.py
+++ b/timeeval_experiments/algorithms/laser_dbn.py
@@ -28,7 +28,7 @@
 }
 
 
-def laser_dbn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def laser_dbn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="LaserDBN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/left_stampi.py b/timeeval_experiments/algorithms/left_stampi.py
index 00bb946e..90ba78ae 100644
--- a/timeeval_experiments/algorithms/left_stampi.py
+++ b/timeeval_experiments/algorithms/left_stampi.py
@@ -50,7 +50,7 @@ def post_left_stampi(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def left_stampi(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def left_stampi(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Left STAMPi",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/lof.py b/timeeval_experiments/algorithms/lof.py
index de1923fb..84431eba 100644
--- a/timeeval_experiments/algorithms/lof.py
+++ b/timeeval_experiments/algorithms/lof.py
@@ -40,7 +40,7 @@
 }
 
 
-def lof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def lof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="LOF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/lstm_ad.py b/timeeval_experiments/algorithms/lstm_ad.py
index d43640cd..ee729f6f 100644
--- a/timeeval_experiments/algorithms/lstm_ad.py
+++ b/timeeval_experiments/algorithms/lstm_ad.py
@@ -92,7 +92,7 @@ def post_lstm_ad(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def lstm_ad(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def lstm_ad(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="LSTM-AD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/lstm_vae.py b/timeeval_experiments/algorithms/lstm_vae.py
index 42db2240..d87a57a2 100644
--- a/timeeval_experiments/algorithms/lstm_vae.py
+++ b/timeeval_experiments/algorithms/lstm_vae.py
@@ -64,7 +64,7 @@
 }
 
 
-def lstm_vae(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def lstm_vae(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="LSTM-VAE",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/median_method.py b/timeeval_experiments/algorithms/median_method.py
index 3a713924..2722fe79 100644
--- a/timeeval_experiments/algorithms/median_method.py
+++ b/timeeval_experiments/algorithms/median_method.py
@@ -22,7 +22,7 @@
 }
 
 
-def median_method(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def median_method(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="MedianMethod",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/mscred.py b/timeeval_experiments/algorithms/mscred.py
index 97be4009..8a71d86a 100644
--- a/timeeval_experiments/algorithms/mscred.py
+++ b/timeeval_experiments/algorithms/mscred.py
@@ -18,7 +18,7 @@ def post_mscred(scores: np.ndarray, args: dict) -> np.ndarray:
     max_window_size = max(args.get("hyper_params", {}).get("windows", [10, 30, 60]))
     offset = (ds_length - (max_window_size - 1)) % gap_time
     image_scores = ReverseWindowing(window_size=window_size).fit_transform(scores)
-    return np.concatenate([np.repeat(image_scores[:-offset], gap_time), image_scores[-offset:]])
+    return np.concatenate([np.repeat(image_scores[:-offset], gap_time), image_scores[-offset:]])  # type: ignore
 
 
 _mscred_parameters: Dict[str, Dict[str, Any]] = {
@@ -95,7 +95,7 @@ def post_mscred(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def mscred(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def mscred(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="MSCRED",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/mstamp.py b/timeeval_experiments/algorithms/mstamp.py
index 19bd8584..dd78ed6f 100644
--- a/timeeval_experiments/algorithms/mstamp.py
+++ b/timeeval_experiments/algorithms/mstamp.py
@@ -31,7 +31,7 @@ def post_mstamp(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def mstamp(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def mstamp(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="mSTAMP",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/mtad_gat.py b/timeeval_experiments/algorithms/mtad_gat.py
index 78e6ec42..3ddc3d64 100644
--- a/timeeval_experiments/algorithms/mtad_gat.py
+++ b/timeeval_experiments/algorithms/mtad_gat.py
@@ -119,7 +119,7 @@ def post_mtad_gat(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def mtad_gat(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def mtad_gat(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="MTAD-GAT",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/multi_hmm.py b/timeeval_experiments/algorithms/multi_hmm.py
index a4ff190b..842f0b84 100644
--- a/timeeval_experiments/algorithms/multi_hmm.py
+++ b/timeeval_experiments/algorithms/multi_hmm.py
@@ -28,7 +28,7 @@
 }
 
 
-def multi_hmm(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def multi_hmm(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="MultiHMM",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/multi_subsequence_lof.py b/timeeval_experiments/algorithms/multi_subsequence_lof.py
index 42607d69..369be9e5 100644
--- a/timeeval_experiments/algorithms/multi_subsequence_lof.py
+++ b/timeeval_experiments/algorithms/multi_subsequence_lof.py
@@ -61,7 +61,7 @@ def post_multi_subsequence_lof(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def multi_subsequence_lof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def multi_subsequence_lof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Multi-Sub-LOF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/multinorma.py b/timeeval_experiments/algorithms/multinorma.py
index c7992050..d46b01be 100644
--- a/timeeval_experiments/algorithms/multinorma.py
+++ b/timeeval_experiments/algorithms/multinorma.py
@@ -58,7 +58,7 @@
 }
 
 
-def multinorma(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def multinorma(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="MultiNormA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/mvalmod.py b/timeeval_experiments/algorithms/mvalmod.py
index 0e6997f6..b14429a5 100644
--- a/timeeval_experiments/algorithms/mvalmod.py
+++ b/timeeval_experiments/algorithms/mvalmod.py
@@ -56,7 +56,7 @@ def post_valmod(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def mvalmod(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def mvalmod(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="mVALMOD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/norma.py b/timeeval_experiments/algorithms/norma.py
index c5e40147..a2b6cc68 100644
--- a/timeeval_experiments/algorithms/norma.py
+++ b/timeeval_experiments/algorithms/norma.py
@@ -38,7 +38,7 @@ def _post_norma(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def norma(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def norma(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="NormA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/normalizing_flows.py b/timeeval_experiments/algorithms/normalizing_flows.py
index 552fd278..684dd3d6 100644
--- a/timeeval_experiments/algorithms/normalizing_flows.py
+++ b/timeeval_experiments/algorithms/normalizing_flows.py
@@ -100,7 +100,7 @@ def post_nf(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def normalizing_flows(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def normalizing_flows(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Normalizing Flows",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/novelty_svr.py b/timeeval_experiments/algorithms/novelty_svr.py
index f2c68e9b..872fe77b 100644
--- a/timeeval_experiments/algorithms/novelty_svr.py
+++ b/timeeval_experiments/algorithms/novelty_svr.py
@@ -106,7 +106,7 @@
 }
 
 
-def novelty_svr(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def novelty_svr(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="NoveltySVR",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/numenta_htm.py b/timeeval_experiments/algorithms/numenta_htm.py
index e6f795bb..fc3770db 100644
--- a/timeeval_experiments/algorithms/numenta_htm.py
+++ b/timeeval_experiments/algorithms/numenta_htm.py
@@ -154,7 +154,7 @@
 }
 
 
-def numenta_htm(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def numenta_htm(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="NumentaHTM",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/ocean_wnn.py b/timeeval_experiments/algorithms/ocean_wnn.py
index c73f41b0..4125df6f 100644
--- a/timeeval_experiments/algorithms/ocean_wnn.py
+++ b/timeeval_experiments/algorithms/ocean_wnn.py
@@ -106,7 +106,7 @@
 }
 
 
-def ocean_wnn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def ocean_wnn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="OceanWNN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/omnianomaly.py b/timeeval_experiments/algorithms/omnianomaly.py
index 72cc1538..e3abd494 100644
--- a/timeeval_experiments/algorithms/omnianomaly.py
+++ b/timeeval_experiments/algorithms/omnianomaly.py
@@ -85,7 +85,7 @@ def post_omni_anomaly(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def omnianomaly(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def omnianomaly(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="OmniAnomaly",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/pcc.py b/timeeval_experiments/algorithms/pcc.py
index 5df84139..37460757 100644
--- a/timeeval_experiments/algorithms/pcc.py
+++ b/timeeval_experiments/algorithms/pcc.py
@@ -52,7 +52,7 @@
 }
 
 
-def pcc(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def pcc(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="PCC",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/pci.py b/timeeval_experiments/algorithms/pci.py
index 08059d90..34022e6c 100644
--- a/timeeval_experiments/algorithms/pci.py
+++ b/timeeval_experiments/algorithms/pci.py
@@ -28,7 +28,7 @@
 }
 
 
-def pci(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def pci(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="PCI",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/phasespace_svm.py b/timeeval_experiments/algorithms/phasespace_svm.py
index a700e7fd..b0241a00 100644
--- a/timeeval_experiments/algorithms/phasespace_svm.py
+++ b/timeeval_experiments/algorithms/phasespace_svm.py
@@ -74,7 +74,7 @@
 }
 
 
-def phasespace_svm(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def phasespace_svm(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="PhaseSpace-SVM",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/pst.py b/timeeval_experiments/algorithms/pst.py
index 95bf6424..ea836627 100644
--- a/timeeval_experiments/algorithms/pst.py
+++ b/timeeval_experiments/algorithms/pst.py
@@ -52,7 +52,7 @@
 }
 
 
-def pst(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def pst(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="PST",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/random_black_forest.py b/timeeval_experiments/algorithms/random_black_forest.py
index ef36a407..8c769440 100644
--- a/timeeval_experiments/algorithms/random_black_forest.py
+++ b/timeeval_experiments/algorithms/random_black_forest.py
@@ -88,7 +88,7 @@
 }
 
 
-def random_black_forest(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def random_black_forest(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Random Black Forest (RR)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/robust_pca.py b/timeeval_experiments/algorithms/robust_pca.py
index 55902fdb..774795d5 100644
--- a/timeeval_experiments/algorithms/robust_pca.py
+++ b/timeeval_experiments/algorithms/robust_pca.py
@@ -22,7 +22,7 @@
 }
 
 
-def robust_pca(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def robust_pca(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="RobustPCA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/s_h_esd.py b/timeeval_experiments/algorithms/s_h_esd.py
index 9dba289c..4438fe61 100644
--- a/timeeval_experiments/algorithms/s_h_esd.py
+++ b/timeeval_experiments/algorithms/s_h_esd.py
@@ -28,7 +28,7 @@
 }
 
 
-def s_h_esd(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def s_h_esd(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="S-H-ESD (Twitter)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/sand.py b/timeeval_experiments/algorithms/sand.py
index 4babb90f..15c12998 100644
--- a/timeeval_experiments/algorithms/sand.py
+++ b/timeeval_experiments/algorithms/sand.py
@@ -52,7 +52,7 @@
 }
 
 
-def sand(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def sand(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="SAND",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/sarima.py b/timeeval_experiments/algorithms/sarima.py
index 29f51bbf..05389696 100644
--- a/timeeval_experiments/algorithms/sarima.py
+++ b/timeeval_experiments/algorithms/sarima.py
@@ -58,7 +58,7 @@
 }
 
 
-def sarima(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def sarima(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="SARIMA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/series2graph.py b/timeeval_experiments/algorithms/series2graph.py
index 0e5a3f9a..335bc303 100644
--- a/timeeval_experiments/algorithms/series2graph.py
+++ b/timeeval_experiments/algorithms/series2graph.py
@@ -46,7 +46,7 @@ def post_s2g(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def series2graph(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def series2graph(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Series2Graph",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/sr.py b/timeeval_experiments/algorithms/sr.py
index 93ebd9d3..85a548ff 100644
--- a/timeeval_experiments/algorithms/sr.py
+++ b/timeeval_experiments/algorithms/sr.py
@@ -34,7 +34,7 @@
 }
 
 
-def sr(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def sr(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Spectral Residual (SR)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/sr_cnn.py b/timeeval_experiments/algorithms/sr_cnn.py
index 445021c7..35f75cc4 100644
--- a/timeeval_experiments/algorithms/sr_cnn.py
+++ b/timeeval_experiments/algorithms/sr_cnn.py
@@ -76,7 +76,7 @@
 }
 
 
-def sr_cnn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def sr_cnn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="SR-CNN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/ssa.py b/timeeval_experiments/algorithms/ssa.py
index 9407b11e..ee2ab9de 100644
--- a/timeeval_experiments/algorithms/ssa.py
+++ b/timeeval_experiments/algorithms/ssa.py
@@ -40,7 +40,7 @@
 }
 
 
-def ssa(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def ssa(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="SSA",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/stamp.py b/timeeval_experiments/algorithms/stamp.py
index e9e6f6d3..792591fa 100644
--- a/timeeval_experiments/algorithms/stamp.py
+++ b/timeeval_experiments/algorithms/stamp.py
@@ -52,7 +52,7 @@ def post_stamp(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def stamp(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def stamp(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="STAMP",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/stomp.py b/timeeval_experiments/algorithms/stomp.py
index 1fef757f..329f8e63 100644
--- a/timeeval_experiments/algorithms/stomp.py
+++ b/timeeval_experiments/algorithms/stomp.py
@@ -58,7 +58,7 @@ def post_stomp(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def stomp(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def stomp(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="STOMP",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/subsequence_fast_mcd.py b/timeeval_experiments/algorithms/subsequence_fast_mcd.py
index 64db5a67..18ce4a7e 100644
--- a/timeeval_experiments/algorithms/subsequence_fast_mcd.py
+++ b/timeeval_experiments/algorithms/subsequence_fast_mcd.py
@@ -37,7 +37,7 @@ def post_sfmcd(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def subsequence_fast_mcd(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def subsequence_fast_mcd(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Subsequence Fast-MCD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/subsequence_if.py b/timeeval_experiments/algorithms/subsequence_if.py
index b50cec1f..510da2aa 100644
--- a/timeeval_experiments/algorithms/subsequence_if.py
+++ b/timeeval_experiments/algorithms/subsequence_if.py
@@ -67,7 +67,7 @@ def post_sIF(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def subsequence_if(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def subsequence_if(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Subsequence IF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/subsequence_knn.py b/timeeval_experiments/algorithms/subsequence_knn.py
index 81e0c085..0568b0ef 100644
--- a/timeeval_experiments/algorithms/subsequence_knn.py
+++ b/timeeval_experiments/algorithms/subsequence_knn.py
@@ -67,7 +67,7 @@ def post_subsequence_knn(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def subsequence_knn(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def subsequence_knn(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Sub-KNN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/subsequence_lof.py b/timeeval_experiments/algorithms/subsequence_lof.py
index 849c2a05..9775fa63 100644
--- a/timeeval_experiments/algorithms/subsequence_lof.py
+++ b/timeeval_experiments/algorithms/subsequence_lof.py
@@ -61,7 +61,7 @@ def post_sLOF(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def subsequence_lof(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def subsequence_lof(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Subsequence LOF",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/subsequence_lof_multi_sum.py b/timeeval_experiments/algorithms/subsequence_lof_multi_sum.py
index 4d30c7dc..59f4e398 100644
--- a/timeeval_experiments/algorithms/subsequence_lof_multi_sum.py
+++ b/timeeval_experiments/algorithms/subsequence_lof_multi_sum.py
@@ -55,7 +55,7 @@ def post_sLOF(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def subsequence_lof_multi_sum(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def subsequence_lof_multi_sum(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Subsequence LOF Multivariate Sum",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/tanogan.py b/timeeval_experiments/algorithms/tanogan.py
index ecf643fb..fa2abe8e 100644
--- a/timeeval_experiments/algorithms/tanogan.py
+++ b/timeeval_experiments/algorithms/tanogan.py
@@ -89,7 +89,7 @@ def post_tanogan(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def tanogan(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def tanogan(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="TAnoGan",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/tarzan.py b/timeeval_experiments/algorithms/tarzan.py
index a634049e..7609d127 100644
--- a/timeeval_experiments/algorithms/tarzan.py
+++ b/timeeval_experiments/algorithms/tarzan.py
@@ -37,7 +37,7 @@ def post_tarzan(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def tarzan(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def tarzan(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="TARZAN",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/telemanom.py b/timeeval_experiments/algorithms/telemanom.py
index 12914923..02e597c6 100644
--- a/timeeval_experiments/algorithms/telemanom.py
+++ b/timeeval_experiments/algorithms/telemanom.py
@@ -103,7 +103,7 @@ def post_telemanom(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def telemanom(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def telemanom(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Telemanom",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/torsk.py b/timeeval_experiments/algorithms/torsk.py
index 93019dba..2ae5acde 100644
--- a/timeeval_experiments/algorithms/torsk.py
+++ b/timeeval_experiments/algorithms/torsk.py
@@ -117,7 +117,7 @@ def _post_torsk(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def torsk(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def torsk(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Torsk",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/triple_es.py b/timeeval_experiments/algorithms/triple_es.py
index e8532ef6..7fa7a52b 100644
--- a/timeeval_experiments/algorithms/triple_es.py
+++ b/timeeval_experiments/algorithms/triple_es.py
@@ -40,7 +40,7 @@
 }
 
 
-def triple_es(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def triple_es(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="Triple ES (Holt-Winter's)",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/ts_bitmap.py b/timeeval_experiments/algorithms/ts_bitmap.py
index eae7a2fe..7a9ed6b7 100644
--- a/timeeval_experiments/algorithms/ts_bitmap.py
+++ b/timeeval_experiments/algorithms/ts_bitmap.py
@@ -52,7 +52,7 @@
 }
 
 
-def ts_bitmap(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def ts_bitmap(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="TSBitmap",
         main=DockerAdapter(
diff --git a/timeeval_experiments/algorithms/valmod.py b/timeeval_experiments/algorithms/valmod.py
index e4fea9f7..164aa366 100644
--- a/timeeval_experiments/algorithms/valmod.py
+++ b/timeeval_experiments/algorithms/valmod.py
@@ -56,7 +56,7 @@ def post_valmod(scores: np.ndarray, args: dict) -> np.ndarray:
 }
 
 
-def valmod(params: ParameterConfig = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
+def valmod(params: Optional[ParameterConfig] = None, skip_pull: bool = False, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="VALMOD",
         main=DockerAdapter(
diff --git a/timeeval_experiments/baselines.py b/timeeval_experiments/baselines.py
index 3fa1803a..6942d5a8 100644
--- a/timeeval_experiments/baselines.py
+++ b/timeeval_experiments/baselines.py
@@ -48,7 +48,8 @@ def fn(X: AlgorithmParameter, params: Dict[str, Any]) -> AlgorithmParameter:
             if isinstance(X, Path):
                 raise ValueError("Increasing baseline requires an np.ndarray as input!")
             indices = np.arange(X.shape[0])
-            return MinMaxScaler().fit_transform(indices.reshape(-1, 1)).reshape(-1)
+            scores: np.ndarray = MinMaxScaler().fit_transform(indices.reshape(-1, 1)).reshape(-1)
+            return scores
 
         return Algorithm(
             name="increasing",
@@ -103,9 +104,9 @@ def _deviating_from(data: np.ndarray, fn: Callable) -> np.ndarray:
         # univariate
         if len(data.shape) == 1:
             diffs = np.abs(data - fn(data))
-            return diffs / diffs.max()
+            return diffs / diffs.max()  # type: ignore
         # multivariate
         else:
             diffs = np.abs(data - fn(data, axis=0))
             diffs = diffs / diffs.max(axis=0)
-            return diffs.mean(axis=1)
+            return diffs.mean(axis=1)  # type: ignore
diff --git a/timeeval_experiments/generator/__main__.py b/timeeval_experiments/generator/__main__.py
index 86520df5..c3716353 100644
--- a/timeeval_experiments/generator/__main__.py
+++ b/timeeval_experiments/generator/__main__.py
@@ -11,9 +11,11 @@
 
 
 def _create_generate_algorithm_stubs_parser(subparsers: Any) -> argparse.ArgumentParser:
-    parser = subparsers.add_parser(ALGO_STUBS_COMMAND,
-                                   help="Generate python stubs for algorithms of the timeeval-algorithms "
-                                        "repository that are implemented as Docker images.")
+    parser: argparse.ArgumentParser = subparsers.add_parser(
+        ALGO_STUBS_COMMAND,
+        help="Generate python stubs for algorithms of the timeeval-algorithms repository that are implemented as "
+             "Docker images."
+    )
     parser.add_argument("repository_path", type=Path,
                         help="Folder containing the timeeval-algorithms repository (source for the Docker images; "
                              "required structure: each algorithm in its own folder, the folder contains a "
@@ -26,8 +28,9 @@ def _create_generate_algorithm_stubs_parser(subparsers: Any) -> argparse.Argumen
 
 
 def _create_generate_param_config_parser(subparsers: Any) -> argparse.ArgumentParser:
-    parser = subparsers.add_parser(PARAM_CONFIG_COMMAND,
-                                   help="Generate parameter configuration (JSON) based on parameter matrix.")
+    parser: argparse.ArgumentParser = subparsers.add_parser(PARAM_CONFIG_COMMAND,
+                                                            help="Generate parameter configuration (JSON) based on "
+                                                                 "parameter matrix.")
     parser.add_argument("parameter_matrix_path", type=Path,
                         help="Path to the file that contains the parameter matrix (direct CSV export of the Google "
                              "Spreadsheet document).")
@@ -80,10 +83,9 @@ def _run_config_gen(args: argparse.Namespace) -> None:
         target=args.output,
         overwrite=args.force
     )
-    pass
 
 
-def main(argv: List[str]):
+def main(argv: List[str]) -> None:
     parser = _create_arg_parser()
     args = parser.parse_args(argv[1:])
     if not args.command:
diff --git a/timeeval_experiments/generator/exceptions.py b/timeeval_experiments/generator/exceptions.py
index 068e6578..bfa3c564 100644
--- a/timeeval_experiments/generator/exceptions.py
+++ b/timeeval_experiments/generator/exceptions.py
@@ -1,23 +1,23 @@
 class AlgorithmManifestLoadingWarning(RuntimeWarning):
-    def __init__(self, msg):
+    def __init__(self, msg: str):
         super().__init__(msg)
 
 
 class MissingManifestWarning(AlgorithmManifestLoadingWarning):
     @staticmethod
-    def msg(name: str):
+    def msg(name: str) -> str:
         return f"Algorithm {name} has no manifest! Skipping {name}."
 
 
 class MissingReadmeWarning(AlgorithmManifestLoadingWarning):
     @staticmethod
-    def msg(name: str):
+    def msg(name: str) -> str:
         return f"Algorithm {name} has no README! Skipping {name}."
 
 
 class InvalidManifestWarning(AlgorithmManifestLoadingWarning):
     @staticmethod
-    def msg(name: str, detail_msg: str = "", will_skip: bool = True):
+    def msg(name: str, detail_msg: str = "", will_skip: bool = True) -> str:
         text = f"Algorithm {name}'s manifest is invalid! {detail_msg}"
         if will_skip:
             text += f"Skipping {name}."
diff --git a/timeeval_experiments/generator/parameter_matrix_parsing.py b/timeeval_experiments/generator/parameter_matrix_parsing.py
index 51022458..b15b7996 100644
--- a/timeeval_experiments/generator/parameter_matrix_parsing.py
+++ b/timeeval_experiments/generator/parameter_matrix_parsing.py
@@ -43,7 +43,7 @@ def _get_params_df(df: pd.DataFrame) -> pd.DataFrame:
         df_params = df.iloc[0:6].T
         headers = df_params.iloc[0]
         df_params = df_params.iloc[1:]
-        df_params.columns = list(map(lambda x: x.lower(), headers))
+        df_params.columns = list(map(lambda x: x.lower(), headers))  # type: ignore
         return df_params
 
     @staticmethod
diff --git a/timeeval_experiments/generator/templates/docker-algorithm.py.jinja b/timeeval_experiments/generator/templates/docker-algorithm.py.jinja
index 6c5b1b2d..bcc72169 100644
--- a/timeeval_experiments/generator/templates/docker-algorithm.py.jinja
+++ b/timeeval_experiments/generator/templates/docker-algorithm.py.jinja
@@ -19,7 +19,7 @@ _{{ image_name }}_parameters: Dict[str, Dict[str, Any]] = {% if parameters is de
 {% endif %}
 
 
-def {{ image_name }}(params: ParameterConfig = None, skip_pull: bool = {{ skip_pull }}, timeout: Optional[Duration] = None) -> Algorithm:
+def {{ image_name }}(params: Optional[ParameterConfig] = None, skip_pull: bool = {{ skip_pull }}, timeout: Optional[Duration] = None) -> Algorithm:
     return Algorithm(
         name="{{ name }}",
         main=DockerAdapter(


Test patch:
diff --git a/.github/workflows/test-pr.yml b/.github/workflows/test-pr.yml
index 4d364bd6..72590f57 100644
--- a/.github/workflows/test-pr.yml
+++ b/.github/workflows/test-pr.yml
@@ -26,7 +26,7 @@ jobs:
       - name: Install CI dependencies
         run: |
           pip install -r requirements.ci
-      - name: Typcheck with mypy
+      - name: Typecheck with mypy
         run: |
           python setup.py typecheck
   test:
@@ -51,9 +51,6 @@ jobs:
     - name: Install CI dependencies
       run: |
         pip install -r requirements.ci
-    - name: Typcheck with mypy
-      run: |
-        python setup.py typecheck
     - name: Test with pytest
       run: |
         python setup.py test
diff --git a/conftest.py b/conftest.py
index 9508efbf..506bc682 100644
--- a/conftest.py
+++ b/conftest.py
@@ -1,8 +1,11 @@
 import pytest
 
 
-options = {"docker": "mark test to run with docker installed",
-           "dask": "mark test to run with dask installed and being able to SSH itself."}
+options = {
+    "docker": "mark test to run with docker installed",
+    "dask": "mark test to run with dask installed and being able to SSH itself.",
+    "optuna": "mark test to run with optuna installed",
+}
 
 
 def pytest_addoption(parser):
diff --git a/tests/__init__.py b/tests/__init__.py
index 758f47de..9b4a926f 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -1,5 +1,3 @@
-from .test_import_data import TestImportData
-from .test_window_utils import TestReverseWindowing
 import unittest
 
 
diff --git a/timeeval/core/__init__.py b/tests/integration/__init__.py
similarity index 100%
rename from timeeval/core/__init__.py
rename to tests/integration/__init__.py
diff --git a/tests/integration/test_optuna_integration.py b/tests/integration/test_optuna_integration.py
new file mode 100644
index 00000000..75e67764
--- /dev/null
+++ b/tests/integration/test_optuna_integration.py
@@ -0,0 +1,198 @@
+import logging
+import tempfile
+import unittest
+from io import StringIO
+from pathlib import Path
+from unittest.mock import patch, MagicMock, Mock
+
+import pytest
+
+
+@pytest.mark.optuna
+class TestOptunaConfiguration(unittest.TestCase):
+    def test_log_level(self):
+        import optuna
+        from timeeval.integration.optuna import OptunaConfiguration
+
+        # test setting log level on init
+        old_logging_level = optuna.logging.get_verbosity()
+        config = OptunaConfiguration(default_storage="journal-file", log_level=logging.DEBUG)
+        new_logging_level = optuna.logging.get_verbosity()
+        self.assertNotEqual(old_logging_level, new_logging_level)
+        self.assertEqual(new_logging_level, logging.DEBUG)
+
+        # test setting log level on change
+        config.log_level = logging.ERROR
+        new_logging_level2 = optuna.logging.get_verbosity()
+        self.assertNotEqual(new_logging_level, new_logging_level2)
+        self.assertEqual(new_logging_level2, logging.ERROR)
+
+    # @pytest.mark.usefixtures("capfd")
+    def test_update_log_stream(self):
+        import optuna
+        from timeeval.integration.optuna import OptunaConfiguration
+
+        # set up logging
+        logfile = StringIO()
+        root_logger = logging.getLogger()
+        root_logger.setLevel(logging.DEBUG)
+        root_logger.addHandler(logging.StreamHandler(logfile))
+
+        # because we test global options, make sure that Optuna defaults are still set:
+        optuna.logging.enable_default_handler()
+        optuna.logging.disable_propagation()
+
+        test_logger = optuna.logging.get_logger("optuna.test-logger")
+
+        # Optuna logs to stderr by default
+        test_logger.error("logging-test")
+        self.assertNotIn("logging-test", logfile.getvalue())
+        # self.assertIn("logging-test", capfd.readouterr().err)
+
+        # within TimeEval, Optuna logs to root logger from logging configuration
+        config = OptunaConfiguration(default_storage="journal-file", log_level=logging.DEBUG)
+        test_logger.info("logging-test2")
+        self.assertIn("logging-test2", logfile.getvalue())
+        # self.assertNotIn("logging-test2", capfd.readouterr().err)
+
+        # test whether changes on the config are reflected
+        config.use_default_logging = True
+        test_logger.error("logging-test3")
+        self.assertNotIn("logging-test3", logfile.getvalue())
+
+
+@pytest.mark.optuna
+class TestOptunaModule(unittest.TestCase):
+
+    def setUp(self) -> None:
+        self.mock_timeeval = Mock(spec=["distributed", "remote", "exps", "results_path"])
+        self.mock_timeeval.distributed = False
+        self.mock_timeeval.remote = Mock(spec=["config", "run_on_scheduler"])
+        self.mock_timeeval.remote.config = Mock(spec=["scheduler_host"])
+        self.mock_timeeval.exps = Mock(algorithms=[])
+        self.full_storage_url = "postgresql://postgres:hairy_bumblebee@test-host:5432/postgres"
+
+    def test_missing_or_wrong_storage(self):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        with self.assertRaises(ValueError):
+            OptunaModule(OptunaConfiguration(default_storage=None))
+
+    @patch("timeeval.integration.optuna.module.OptunaModule._check_docker_available")
+    def test_checks_docker(self, mock_check):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        OptunaModule(OptunaConfiguration(default_storage="journal-file", dashboard=True))
+        mock_check.assert_called_with("start the Optuna dashboard")
+
+        OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=False))
+        mock_check.assert_called_with("start the Optuna storage postgresql")
+
+    @patch("socket.gethostname")
+    @patch("timeeval.integration.optuna.module._start_postgres_container")
+    @patch("timeeval.integration.optuna.module.OptunaModule._check_docker_available")
+    def test_prepare_start_storage_container(self, mock_check, mock_start_postgres, mock_hostname):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=False))
+        mock_hostname.return_value = "test-host"
+        self.mock_timeeval.distributed = False
+
+        module.prepare(self.mock_timeeval)
+
+        self.assertEqual(module.storage_url, self.full_storage_url)
+        mock_check.assert_called_once()
+        mock_start_postgres.assert_called_with(password="hairy_bumblebee", port=5432)
+
+    @patch("socket.gethostname")
+    @patch("timeeval.integration.optuna.module.OptunaModule._check_docker_available")
+    def test_prepare_not_starting_dashboard(self, mock_check, mock_hostname):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="journal-file", dashboard=True))
+        mock_hostname.return_value = "test-host"
+        self.mock_timeeval.distributed = False
+
+        with tempfile.TemporaryDirectory() as tempdir:
+            self.mock_timeeval.results_path = Path(tempdir)
+            with self.assertLogs(level="WARNING") as logs:
+                module.prepare(self.mock_timeeval)
+            self.assertRegex("\n".join(logs.output), "[C|c]ould not find dashboard connection URL")
+
+        mock_check.assert_called_once()
+        self.assertIsNone(module.storage_url)
+        # mock_start_dashboard.assert_called_with(storage="http://localhost:8080")
+
+    @patch("socket.gethostname")
+    @patch("timeeval.integration.optuna.module._start_dashboard_container")
+    @patch("timeeval.integration.optuna.module._start_postgres_container")
+    @patch("timeeval.integration.optuna.module.OptunaModule._check_docker_available")
+    def test_prepare_start_storage_and_dashboard_container(self, mock_check, mock_start_postgres, mock_start_dashboard, mock_hostname):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=True))
+        mock_hostname.return_value = "test-host"
+        self.mock_timeeval.distributed = False
+
+        module.prepare(self.mock_timeeval)
+
+        self.assertTrue(callable(module.config.default_storage))
+        self.assertEqual(module.storage_url, self.full_storage_url)
+        mock_check.assert_called_once()
+        mock_start_postgres.assert_called_with(password="hairy_bumblebee", port=5432)
+        mock_start_dashboard.assert_called_with(storage=self.full_storage_url)
+
+    @patch("socket.gethostname")
+    @patch("timeeval.integration.optuna.module._start_postgres_container")
+    @patch("timeeval.integration.optuna.module.OptunaModule._check_docker_available")
+    def test_prepare_start_on_scheduler(self, mock_check, mock_start_postgres, mock_hostname):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=False))
+        self.mock_timeeval.distributed = True
+
+        module.prepare(self.mock_timeeval)
+
+        self.mock_timeeval.remote.run_on_scheduler.assert_called_once_with(
+            [(mock_start_postgres, [], {"password": "hairy_bumblebee", "port": 5432})],
+            msg="Starting Optuna containers on scheduler"
+        )
+
+    @patch("timeeval.integration.optuna.module._stop_containers")
+    def test_finalize_local(self, mock_stop_containers):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=False))
+        self.mock_timeeval.distributed = False
+
+        # without remove
+        module.finalize(self.mock_timeeval)
+        mock_stop_containers.assert_called_with(remove=False)
+        self.mock_timeeval.remote.run_on_scheduler.assert_not_called()
+
+        # with remove
+        module.config.remove_managed_containers = True
+        module.finalize(self.mock_timeeval)
+        mock_stop_containers.assert_called_with(remove=True)
+
+    @patch("timeeval.integration.optuna.module._stop_containers")
+    def test_finalize_distributed(self, mock_stop_containers):
+        from timeeval.integration.optuna import OptunaModule, OptunaConfiguration
+
+        module = OptunaModule(OptunaConfiguration(default_storage="postgresql", dashboard=False))
+        self.mock_timeeval.distributed = True
+
+        # without remove
+        module.finalize(self.mock_timeeval)
+        self.mock_timeeval.remote.run_on_scheduler.assert_called_with(
+            [(mock_stop_containers, [], {"remove": False})],
+            msg="Stopping Optuna containers on scheduler"
+        )
+
+        # with remove
+        module.config.remove_managed_containers = True
+        module.finalize(self.mock_timeeval)
+        self.mock_timeeval.remote.run_on_scheduler.assert_called_with(
+            [(mock_stop_containers, [], {"remove": True})],
+            msg="Stopping Optuna containers on scheduler"
+        )
diff --git a/tests/params/test_bayesian.py b/tests/params/test_bayesian.py
new file mode 100644
index 00000000..9f5e8314
--- /dev/null
+++ b/tests/params/test_bayesian.py
@@ -0,0 +1,170 @@
+import socket
+import tempfile
+import unittest
+from pathlib import Path
+
+import optuna
+import pytest
+from optuna.samplers import TPESampler
+from optuna.storages import JournalStorage, JournalFileOpenLock, JournalFileStorage
+
+from timeeval import Algorithm, TrainingType
+from timeeval.adapters import FunctionAdapter
+from timeeval.datasets import Dataset
+from timeeval.integration.optuna import OptunaLazyParams
+from timeeval.metrics import RangePrAUC
+from timeeval.params import BayesianParameterSearch
+
+
+@pytest.mark.optuna
+class TestBayesianParameterSearch(unittest.TestCase):
+
+    def setUp(self) -> None:
+        from optuna import distributions
+
+        self.algorithm = Algorithm(
+            name="TestAlgorithm",
+            main=FunctionAdapter.identity()
+        )
+        self.dataset = Dataset(
+            datasetId=("test", "dataset-name"),
+            dataset_type="synthetic",
+            training_type=TrainingType.UNSUPERVISED,
+            length=50000,
+            dimensions=1,
+            contamination=0.05,
+            min_anomaly_length=20,
+            median_anomaly_length=52,
+            max_anomaly_length=112,
+            period_size=32,
+            num_anomalies=6,
+        )
+        self.param_distributions = {
+            "method": distributions.CategoricalDistribution(["good", "bad"]),
+            "max_features": distributions.FloatDistribution(low=0.0, high=1.0, step=0.01),
+            "window_size": distributions.IntDistribution(low=1, high=1000, step=5),
+        }
+        self.expected_params = [
+            {"method": "bad", "max_features": 0.73, "window_size": 596},
+            {"method": "good", "max_features": 0.05, "window_size": 866},
+        ]
+
+    def test_update_options(self):
+        from timeeval.integration.optuna import OptunaConfiguration, OptunaStudyConfiguration
+        from optuna.samplers import CmaEsSampler, TPESampler
+
+        default_sampler = TPESampler()
+        study_sampler = CmaEsSampler(with_margin=True)
+        metric = RangePrAUC()
+        global_config = OptunaConfiguration(
+            default_storage="sqlite:///optuna.db",
+            default_sampler=default_sampler,
+            continue_existing_studies=True,
+            dashboard=True
+        )
+        optuna_study_config = OptunaStudyConfiguration(
+            n_trials=4,
+            sampler=study_sampler,
+            metric=metric,
+            direction="maximize",
+        )
+        self.assertEqual(optuna_study_config.n_trials, 4)
+        self.assertIsNone(optuna_study_config.storage)
+        self.assertEqual(optuna_study_config.sampler, study_sampler)
+        self.assertIsNone(optuna_study_config.pruner)
+        self.assertEqual(optuna_study_config.metric, metric)
+        self.assertEqual(optuna_study_config.direction, "maximize")
+        self.assertFalse(optuna_study_config.continue_existing_study)
+
+        optuna_study_config = optuna_study_config.update_unset_options(global_config)
+        self.assertEqual(optuna_study_config.n_trials, 4)
+        self.assertEqual(optuna_study_config.storage, "sqlite:///optuna.db")
+        self.assertEqual(optuna_study_config.sampler, study_sampler)
+        self.assertIsNone(optuna_study_config.pruner)
+        self.assertEqual(optuna_study_config.metric, metric)
+        self.assertEqual(optuna_study_config.direction, "maximize")
+        self.assertTrue(optuna_study_config.continue_existing_study)
+
+    def test_lazy_param_creation(self):
+        from timeeval.integration.optuna import OptunaStudyConfiguration
+
+        optuna_study_config = OptunaStudyConfiguration(n_trials=4, metric=RangePrAUC(), storage=None)
+        param_search = BayesianParameterSearch(config=optuna_study_config, params=self.param_distributions)
+        params = list(param_search.iter(self.algorithm, self.dataset))
+        self.assertEqual(len(params), 4)
+
+        for i, param in enumerate(params):
+            trial_id = f"{self.algorithm.name}-{self.dataset.name}-{i}"
+            self.assertIsInstance(param, OptunaLazyParams)
+            self.assertEqual(len(param), len(self.param_distributions))
+            self.assertListEqual(list(param), list(self.param_distributions.keys()))
+            self.assertListEqual(
+                list(param.items()),
+                list(map(lambda k: (k, trial_id), self.param_distributions.keys()))
+            )
+            self.assertEqual(param.uid(), trial_id)
+
+    def test_materialization(self):
+        from timeeval.integration.optuna import OptunaStudyConfiguration
+
+        study_name = f"{self.algorithm.name}-{self.dataset.name}"
+
+        with tempfile.TemporaryDirectory() as tmp_path:
+            tmp_path = Path(tmp_path)
+            journal_file_path = str(tmp_path / "test_materialization.optuna-journal.log")
+            optuna_study_config = OptunaStudyConfiguration(
+                n_trials=4,
+                metric=RangePrAUC(),
+                storage=lambda: JournalStorage(
+                    JournalFileStorage(journal_file_path, lock_obj=JournalFileOpenLock(journal_file_path))
+                ),
+                sampler=TPESampler(seed=42)
+            )
+            param_search = BayesianParameterSearch(config=optuna_study_config, params=self.param_distributions)
+            params = list(param_search.iter(self.algorithm, self.dataset))
+            loaded_study = optuna.load_study(study_name, storage=optuna_study_config.storage())
+            self.assertEqual(loaded_study.study_name, study_name)
+            self.assertEqual(loaded_study.user_attrs["algorithm"], self.algorithm.name)
+            self.assertEqual(loaded_study.user_attrs["dataset"], self.dataset.name)
+            self.assertEqual(loaded_study.user_attrs["metric"], optuna_study_config.metric.name)
+            self.assertEqual(loaded_study.user_attrs["direction"], str(optuna_study_config.direction).lower())
+
+            for i, (param, p_suggested) in enumerate(zip(params, self.expected_params)):
+                trial_id = f"{self.algorithm.name}-{self.dataset.name}-{i}"
+                self.assertIsInstance(param, OptunaLazyParams)
+                param.materialize()
+
+                self.assertIsNotNone(param._study)
+                self.assertIsNotNone(param._trial)
+                self.assertEqual(param.trial().user_attrs["uid"], trial_id)
+                self.assertEqual(param.trial().user_attrs["node"], socket.gethostname())
+
+                self.assertIsInstance(param["method"], str)
+                self.assertIsInstance(param["max_features"], float)
+                self.assertIsInstance(param["window_size"], int)
+                self.assertListEqual(list(param), list(p_suggested))
+                # TimeEval ignores seeding on purpose, so we can't check for the actual values
+                # self.assertEqual(param["method"], p_suggested["method"])
+                # self.assertEqual(param["max_features"], p_suggested["max_features"])
+                # self.assertEqual(param["window_size"], p_suggested["window_size"])
+                # self.assertDictEqual(param.to_dict(), p_suggested)
+
+    def test_empty_params(self):
+        from timeeval.integration.optuna import OptunaStudyConfiguration
+
+        with tempfile.TemporaryDirectory() as tmp_path:
+            tmp_path = Path(tmp_path)
+            journal_file_path = str(tmp_path / "test_empty_params.optuna-journal.log")
+            optuna_study_config = OptunaStudyConfiguration(
+                n_trials=4,
+                metric=RangePrAUC(),
+                storage=lambda: JournalStorage(
+                    JournalFileStorage(journal_file_path, lock_obj=JournalFileOpenLock(journal_file_path))
+                ),
+                sampler=TPESampler(seed=42)
+            )
+            param_search = BayesianParameterSearch(config=optuna_study_config, params={})
+            params = list(param_search.iter(self.algorithm, self.dataset))
+            for p in params:
+                p.materialize()
+                self.assertDictEqual(p.to_dict(), {})
diff --git a/tests/params/test_fixed.py b/tests/params/test_fixed.py
new file mode 100644
index 00000000..70f92289
--- /dev/null
+++ b/tests/params/test_fixed.py
@@ -0,0 +1,23 @@
+import unittest
+
+from timeeval.params import FixedParameters
+from timeeval.params.params import FixedParams
+
+
+class TestFixedParameters(unittest.TestCase):
+
+    def setUp(self) -> None:
+        self.params = {"a": 2, "b": False, "c": "auto"}
+
+    def test_fixed_parameters(self):
+        grid = FixedParameters(self.params)
+        self.assertListEqual(list(grid), [FixedParams(self.params)])
+
+    def test_fixed_parameters_wrong_input(self):
+        with self.assertRaises(TypeError) as ex:
+            FixedParameters([{"a": [1, 2]}])
+        self.assertRegex(str(ex.exception), "[P|p]lease use a.*IndependentParameterGrid.")
+
+        with self.assertRaises(TypeError) as ex:
+            FixedParameters("a")
+        self.assertRegex(str(ex.exception), "[P|p]arameters are not provided as a dict")
diff --git a/tests/params/test_search.py b/tests/params/test_grid_search.py
similarity index 95%
rename from tests/params/test_search.py
rename to tests/params/test_grid_search.py
index f00b1052..b2ebe904 100644
--- a/tests/params/test_search.py
+++ b/tests/params/test_grid_search.py
@@ -1,6 +1,7 @@
 import unittest
 
 from timeeval.params import FullParameterGrid, IndependentParameterGrid
+from timeeval.params.params import FixedParams
 
 
 class TestParameterSearch(unittest.TestCase):
@@ -20,7 +21,7 @@ def test_full_parameter_grid(self):
             {"a": 2, "b": False, "c": "auto"},
             {"a": 2, "b": False, "c": "tanh"}
         ])
-        self.assertEqual(grid[1], {"a": 1, "b": True, "c": "tanh"})
+        self.assertEqual(grid[1], FixedParams({"a": 1, "b": True, "c": "tanh"}))
 
     def test_full_parameter_grid_wrong_input(self):
         with self.assertRaises(TypeError) as ex:
diff --git a/tests/test_algorithm_timer.py b/tests/test_algorithm_timer.py
index b854e7f5..4f32d09f 100644
--- a/tests/test_algorithm_timer.py
+++ b/tests/test_algorithm_timer.py
@@ -6,7 +6,7 @@
 from timeeval import Algorithm, AlgorithmParameter, TrainingType
 from timeeval.adapters import FunctionAdapter
 from timeeval.data_types import ExecutionType
-from timeeval.core.times import Times
+from timeeval._core.times import Times
 
 
 def pre(x: AlgorithmParameter, args) -> AlgorithmParameter:
diff --git a/tests/test_dataset_and_algorithm_match.py b/tests/test_dataset_and_algorithm_match.py
index 22c9e9ac..7ccf5e0d 100644
--- a/tests/test_dataset_and_algorithm_match.py
+++ b/tests/test_dataset_and_algorithm_match.py
@@ -18,7 +18,8 @@
     DatasetManager
 )
 from timeeval.datasets import Dataset, DatasetRecord
-from timeeval.core.experiments import Experiment, Experiments
+from timeeval._core.experiments import Experiment, Experiments
+from timeeval.params.params import FixedParams
 from timeeval.utils.hash_dict import hash_dict
 
 
@@ -146,7 +147,7 @@ def test_missing_training_dataset_experiment(self):
                 period_size=None
             ),
             algorithm=self.algorithms[0],
-            params={},
+            params=FixedParams({}),
             params_id=hash_dict({}),
             repetition=0,
             base_results_dir=Path("tmp_path"),
diff --git a/tests/test_distributed_timeeval.py b/tests/test_distributed_timeeval.py
index 17749353..1772bd2b 100644
--- a/tests/test_distributed_timeeval.py
+++ b/tests/test_distributed_timeeval.py
@@ -73,11 +73,11 @@ def test_distributed_results_and_shutdown_cluster(self):
                     f"{[p.cmdline() for p in processes]}"
             )
 
-    @patch("timeeval.core.remote.Popen")
+    @patch("timeeval._core.remote.Popen")
     @patch("timeeval.timeeval.subprocess.call")
     @patch("timeeval.adapters.docker.docker.from_env")
-    @patch("timeeval.core.remote.Client")
-    @patch("timeeval.core.remote.SSHCluster")
+    @patch("timeeval._core.remote.Client")
+    @patch("timeeval._core.remote.SSHCluster")
     def test_distributed_phases(self, mock_cluster, mock_client, mock_docker, mock_call, mock_popen):
         mock_client.return_value = MockDaskClient()
         mock_cluster.return_value = MockDaskSSHCluster(workers=2)
@@ -124,11 +124,11 @@ def test_phases(self, mock_docker):
                 (timeeval.results_path / "docker" / hash_dict({}) / "custom" / "dataset.1" / "1").exists()
             )
 
-    @patch("timeeval.core.remote.Popen")
+    @patch("timeeval._core.remote.Popen")
     @patch("timeeval.timeeval.subprocess.call")
     @patch("timeeval.adapters.docker.docker.from_env")
-    @patch("timeeval.core.remote.Client")
-    @patch("timeeval.core.remote.SSHCluster")
+    @patch("timeeval._core.remote.Client")
+    @patch("timeeval._core.remote.SSHCluster")
     def test_aliases_excluded(self, mock_cluster, mock_client, mock_docker, mock_call, mock_popen):
         mock_client.return_value = MockDaskClient()
         mock_cluster.return_value = MockDaskSSHCluster(workers=2)
@@ -151,11 +151,11 @@ def test_aliases_excluded(self, mock_cluster, mock_client, mock_docker, mock_cal
             self.assertEqual(len(rsync.params), 1)
             self.assertTrue(rsync.params[0], ["rsync", "-a", f"test-host:{tmp_path}/", tmp_path])
 
-    @patch("timeeval.core.remote.Popen")
+    @patch("timeeval._core.remote.Popen")
     @patch("timeeval.timeeval.subprocess.call")
     @patch("timeeval.adapters.docker.docker.from_env")
-    @patch("timeeval.core.remote.Client")
-    @patch("timeeval.core.remote.SSHCluster")
+    @patch("timeeval._core.remote.Client")
+    @patch("timeeval._core.remote.SSHCluster")
     def test_catches_future_exception(self, mock_cluster, mock_client, mock_docker, mock_call, mock_popen):
         mock_client.return_value = MockDaskExceptionClient()
         mock_cluster.return_value = MockDaskSSHCluster(workers=2)
@@ -175,11 +175,11 @@ def test_catches_future_exception(self, mock_cluster, mock_client, mock_docker,
             timeeval.run()
             self.assertListEqual(timeeval.results[["status", "error_message"]].values[0].tolist(), [Status.ERROR, "ExceptionForTest('test-exception')"])
 
-    @patch("timeeval.core.remote.Popen")
+    @patch("timeeval._core.remote.Popen")
     @patch("timeeval.timeeval.subprocess.call")
     @patch("timeeval.adapters.docker.docker.from_env")
-    @patch("timeeval.core.remote.Client")
-    @patch("timeeval.core.remote.SSHCluster")
+    @patch("timeeval._core.remote.Client")
+    @patch("timeeval._core.remote.SSHCluster")
     def test_catches_future_timeout_exception(self, mock_cluster, mock_client, mock_docker, mock_call, mock_popen):
         mock_client.return_value = MockDaskDockerTimeoutExceptionClient()
         mock_cluster.return_value = MockDaskSSHCluster(workers=2)
@@ -200,11 +200,11 @@ def test_catches_future_timeout_exception(self, mock_cluster, mock_client, mock_
             self.assertListEqual(timeeval.results[["status", "error_message"]].values[0].tolist(),
                                  [Status.TIMEOUT, "DockerTimeoutError('test-exception-timeout')"])
 
-    @patch("timeeval.core.remote.Popen")
+    @patch("timeeval._core.remote.Popen")
     @patch("timeeval.timeeval.subprocess.call")
     @patch("timeeval.adapters.docker.docker.from_env")
-    @patch("timeeval.core.remote.Client")
-    @patch("timeeval.core.remote.SSHCluster")
+    @patch("timeeval._core.remote.Client")
+    @patch("timeeval._core.remote.SSHCluster")
     def test_catches_future_memory_exception(self, mock_cluster, mock_client, mock_docker, mock_call, mock_popen):
         mock_client.return_value = MockDaskDockerMemoryExceptionClient()
         mock_cluster.return_value = MockDaskSSHCluster(workers=2)
diff --git a/tests/test_experiments.py b/tests/test_experiments.py
index c920e766..bf855c70 100755
--- a/tests/test_experiments.py
+++ b/tests/test_experiments.py
@@ -4,7 +4,7 @@
 
 from tests.fixtures.algorithms import SupervisedDeviatingFromMean
 from timeeval import Algorithm, TrainingType, InputDimensionality, DefaultMetrics, DatasetManager
-from timeeval.core.experiments import Experiments
+from timeeval._core.experiments import Experiments
 from timeeval.params import FullParameterGrid
 
 
@@ -64,13 +64,13 @@ def test_common_params_id_for_heuristics(self):
         tasks = [(exp.dataset_name, exp.params) for exp in exps]
         self.assertListEqual(tasks, [
             ("dataset-datetime", {"param1": 0, "param2":  12}),
-            ("dataset-int",      {"param1": 0, "param2": 100}),
             ("dataset-datetime", {"param1": 1, "param2":  12}),
+            ("dataset-int",      {"param1": 0, "param2": 100}),
             ("dataset-int",      {"param1": 1, "param2": 100})
         ])
 
         # the parameter IDs for both datasets must be the same despite having different values in param2, because
         # param2 is set by a heuristic depending on the dataset's metadata
         params_ids = [exp.params_id for exp in exps]
-        self.assertEqual(params_ids[0], params_ids[1])
-        self.assertEqual(params_ids[2], params_ids[3])
+        self.assertEqual(params_ids[0], params_ids[2])
+        self.assertEqual(params_ids[1], params_ids[3])
diff --git a/tests/test_remote.py b/tests/test_remote.py
index f1eb3126..41d97ee8 100644
--- a/tests/test_remote.py
+++ b/tests/test_remote.py
@@ -8,7 +8,7 @@
 import pytest
 from dask.distributed import SSHCluster
 
-from timeeval.core.remote import Remote, RemoteConfiguration
+from timeeval._core.remote import Remote, RemoteConfiguration
 
 
 class TestRemote(unittest.TestCase):
diff --git a/tests/test_timeeval_exceptions.py b/tests/test_timeeval_exceptions.py
index 653c8db9..90e1a746 100644
--- a/tests/test_timeeval_exceptions.py
+++ b/tests/test_timeeval_exceptions.py
@@ -17,7 +17,7 @@ def setUp(self) -> None:
         self.datasets: Datasets = DatasetManager("./tests/example_data", custom_datasets_file=self.datasets_config)
         self.identity_algorithm = Algorithm(name="test", main=FunctionAdapter.identity(), data_as_file=False)
 
-    @patch("timeeval.core.experiments.load_dataset")
+    @patch("timeeval._core.experiments.load_dataset")
     def test_wrong_df_shape(self, mock_load):
         df = pd.DataFrame(np.random.rand(10, 2))
         mock_load.side_effect = [df]