Repository: automl/DeepCAVE (language: python)

Issue:
Typehints, Doc and tests



Discussion / hints:

Fix patch:
diff --git a/.DS_Store b/.DS_Store
deleted file mode 100644
index a7ca1320..00000000
Binary files a/.DS_Store and /dev/null differ
diff --git a/.flake8 b/.flake8
new file mode 100644
index 00000000..f8c018cf
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,12 @@
+[flake8]
+show-source = True
+max-line-length = 100
+extend-exclude =
+    venv
+    .venv
+    build
+extend-ignore =
+    # No whitespace before ':' in [x : y]
+    E203
+    # No lambdas — too strict
+    E731
\ No newline at end of file
diff --git a/.gitignore b/.gitignore
index ed01a9cc..7f91e220 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,12 +2,22 @@ build/*
 *cache*
 !*cache.py
 !*caches.py
-docs/*
 __pycache__
 *.rdb
 examples/logs/SMAC
 examples/logs/BOHB
 smac3-output*
 .DS_Store
-pending
-__init__.pyc
\ No newline at end of file
+__init__.pyc
+examples/logs*
+examples/*
+!examples/logs/DeepCAVE
+!examples/record
+examples/record/_*.py
+*.sh
+!run.sh
+.vscode
+*.egg-info
+docs/apidoc
+docs/html
+docs/pages/examples
\ No newline at end of file
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 00000000..8c6e34e1
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,59 @@
+repos:
+
+  - repo: https://github.com/pycqa/isort
+    rev: 5.10.1
+    hooks:
+      - id: isort
+        name: isort imports deepcave
+        files: deepcave/.*
+        args: [--check]
+
+      - id: isort
+        name: isort imports test
+        files: test/.*
+        args: [--check]
+
+  - repo: https://github.com/ambv/black
+    rev: 21.12b0
+    hooks:
+      - id: black
+        name: black formatter deepcave
+        files: deepcave/.*
+        args: [--check]
+
+      - id: black
+        name: black formatter test
+        files: test/.*
+        args: [--check]
+
+      - id: black
+        name: black formatter examples
+        files: examples/.*
+        args: [--check]
+
+  # This is disabled as most modules fail this
+  - repo: https://github.com/pycqa/pydocstyle
+    rev: 6.1.1
+    hooks:
+      - id: pydocstyle
+        files: DISABLED  # deepcave/.*
+        always_run: false
+        additional_dependencies: ["toml"] # Needed to parse pyproject.toml
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.930
+    hooks:
+      - id: mypy
+        name: mypy deepcave
+        files: deepcave/.*
+
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: 4.0.1
+    hooks:
+      - id: flake8
+        name: flake8 deepcave
+        files: deepcave/.*
+
+      - id: flake8
+        name: flake8 test
+        files: test/.*
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fa035e2b..7e4190bb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,41 @@
+# 0.2
+
+## Runs
+- Groups were replaced by `GroupedRun`. Hence, a group can be used as a normal run (`AbstractRun`).
+- Groups are only created if the runs are compatible (via function `check_equality`).
+
+## Plugins
+- Integrated Parallel Coordinates.
+- Configurations Cube and Parallel Coordinates show the real config values and not
+the encoded anymore.
+- Changed plugin interfaces (check_run_compatibility, check_runs_compatibility,
+load_inputs, load_dependency_inputs).
+- Enhanced native group selection drastically.
+- Added icons to the plugins.
+- Removed ICE for now as too many bugs were found.
+
+## Others
+- Added typehints.
+- Added tests for caches, basic layouts and other utils.
+- Added tools (mypy, flake8 and blake)
+- Performance increase in `general` as a folder is only transformed to a run if selected.
+
+## Known Bugs
+- fANOVA does not work if the configspace includes constants.
+
+
+# 0.1.1
+
+## Plugins
+
+- ICE/PDP: Added uncertainties
+- fANOVA: Works with no budgets now
+
+## Others
+
+- Installable via pip
+
+
 # 0.1
 
 ## Features
@@ -8,7 +46,6 @@
 - Recorder: Record your runs and save them directly as DeepCAVE files.
 - Matplotlib mode: Alternatively, plugins can output Matplotlib rendered plots.
 
-
 ## Plugins
 
 - StaticPlugin: Uses queue to process the result.
@@ -21,9 +58,8 @@
 - Individual Conditional Expectation / Partial Dependency Plots: Shows how individual instances behave.
 - fANOVA: Shows Hyperparameter Importance.
 
-
 ## Converters
 
 - DeepCAVE
 - SMAC
-- BOHB (Beta)
\ No newline at end of file
+- BOHB (Beta)
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 00000000..8d1cd958
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,24 @@
+# syntax=docker/dockerfile:1
+
+FROM continuumio/miniconda3
+
+# Install linux dependencies
+RUN apt-get update -y
+RUN apt install -y build-essential
+RUN apt-get install -y swig
+RUN apt-get install -y redis-server
+
+# Copy files
+COPY . /DeepCAVE
+WORKDIR /DeepCAVE
+
+RUN conda update conda -y
+
+# Create new environment
+RUN conda env create -f environment.yml
+
+# Make RUN commands use the new environment:
+SHELL ["conda", "run", "-n", "DeepCAVE", "/bin/bash", "-c"]
+
+# Install DeepCAVE
+RUN pip install .
\ No newline at end of file
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 00000000..cdeb62aa
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,4 @@
+include requirements.txt
+include extras_require.json
+include loggin.yml
+include start.sh
\ No newline at end of file
diff --git a/Makefile b/Makefile
index 7cd45467..cabe243b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,100 @@
-install:
-	pip install -r requirements.txt
+# NOTE: Used on linux, limited support outside of Linux
+#
+# A simple makefile to help with small tasks related to development of deepcave
+# These have been configured to only really run short tasks. Longer form tasks
+# are usually completed in github actions.
 
+.PHONY: help install-dev check format pre-commit clean clean-build build publish test
+
+help:
+	@echo "Makefile deepcave"
+	@echo "* install-dev      to install all dev requirements and install pre-commit"
+	@echo "* check            to check the source code for issues"
+	@echo "* format           to format the code with black and isort"
+	@echo "* pre-commit       to run the pre-commit check"
+	@echo "* clean            to clean the dist and doc build files"
+	@echo "* build            to build a dist"
+	@echo "* publish          to help publish the current branch to pypi"
+	@echo "* tests            to run the tests"
+
+PYTHON ?= python
+CYTHON ?= cython
+PYTEST ?= python -m pytest
+CTAGS ?= ctags
+PIP ?= python -m pip
+MAKE ?= make
+BLACK ?= black
+ISORT ?= isort
+PYDOCSTYLE ?= pydocstyle
+MYPY ?= mypy
+PRECOMMIT ?= pre-commit
+FLAKE8 ?= flake8
+
+DIR := ${CURDIR}
+DIST := ${CURDIR}/dist
+
+install-dev:
+	$(PIP) install -e ".[dev]"
+	pre-commit install
+
+check-black:
+	$(BLACK) deepcave examples tests --check || :
+
+check-isort:
+	$(ISORT) deepcave tests --check || :
+
+check-pydocstyle:
+	$(PYDOCSTYLE) deepcave || :
+
+check-mypy:
+	$(MYPY) deepcave || :
+
+check-flake8:
+	$(FLAKE8) deepcave || :
+	$(FLAKE8) tests || :
+
+# pydocstyle does not have easy ignore rules, instead, we include as they are covered
+check: check-black check-isort check-mypy check-flake8 # check-pydocstyle
+
+pre-commit:
+	$(PRECOMMIT) run --all-files
+
+format-black:
+	$(BLACK) deepcave tests examples
+
+format-isort:
+	$(ISORT) deepcave tests
+
+format: format-black format-isort
+
+clean-build:
+	$(PYTHON) setup.py clean
+	rm -rf ${DIST}
+
+# Clean up any builds in ./dist as well as doc
 clean:
-	rm -rf cache
\ No newline at end of file
+	clean-build
+	rm -rf cache
+
+# Build a distribution in ./dist
+build:
+	$(PYTHON) setup.py bdist
+
+# Publish to testpypi
+# Will echo the commands to actually publish to be run to publish to actual PyPi
+# This is done to prevent accidental publishing but provide the same conveniences
+publish: clean-build build
+	$(PIP) install twine
+	$(PYTHON) -m twine upload --repository testpypi ${DIST}/*
+	@echo
+	@echo "Test with the following line:"
+	@echo "pip install --index-url https://test.pypi.org/simple/ auto-sklearn"
+	@echo
+	@echo "Once you have decided it works, publish to actual pypi with"
+	@echo "python -m twine upload dist/*"
+
+tests:
+	$(PYTEST) tests
+
+
+	
\ No newline at end of file
diff --git a/README.md b/README.md
index 49256216..1ddcf33f 100644
--- a/README.md
+++ b/README.md
@@ -7,16 +7,25 @@ DeepCAVE has two main contributions:
 
 ## Installation
 
+First, make sure you have
+[swig](https://www.dev2qa.com/how-to-install-swig-on-macos-linux-and-windows/) and
+[redis-server](https://flaviocopes.com/redis-installation/) installed on your
+computer.
+
+If you are on an Non-Intel Mac you have to add
 ```
-git clone https://github.com/automl/DeepCAVE.git
-cd DeepCAVE
-conda create -n DeepCAVE python=3.9
-make install
+export DISABLE_SPRING=true
+export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
 ```
+to your ```~/.bash_profile``` to enable multi-processing.
 
-If you want to use DeepCAVE in a different directory set your PYTHONPATH:
+Afterwards, follow the instructions:
 ```
-export PYTHONPATH=$(pwd)
+git clone https://github.com/automl/DeepCAVE.git
+cd DeepCAVE
+conda env create -f environment.yml
+conda activate DeepCAVE
+pip install .
 ```
 
 
@@ -26,7 +35,7 @@ In the following, a minimal example is given to show the simplicity yet powerful
 
 ```
 import ConfigSpace as CS
-from deep_cave import Recorder
+from deep_cave import Recorder, Objective
 
 
 configspace = CS.ConfigurationSpace(seed=0)
@@ -34,7 +43,10 @@ alpha = CS.hyperparameters.UniformFloatHyperparameter(
     name='alpha', lower=0, upper=1)
 configspace.add_hyperparameter(alpha)
 
-with Recorder(configspace, objectives=["accuracy", "mse"]) as r:
+accuracy = Objective("accuracy", lower=0, upper=1, optimize="upper")
+mse = Objective("mse", lower=0)
+
+with Recorder(configspace, objectives=[accuracy, mse]) as r:
     for config in configspace.sample_configuration(100):
         for budget in [20, 40, 60]:
             r.start(config, budget)
@@ -45,8 +57,16 @@ with Recorder(configspace, objectives=["accuracy", "mse"]) as r:
 
 ## Visualizing and Evaluating
 
-The webserver as well as the queue/workers can be started by running ``` ./run.sh ```. 
+The webserver as well as the queue/workers can be started by running
+```
+deepcave --start
+```
+or
+```
+./start.sh
+```
+
 Visit `http://127.0.0.1:8050/` to get started.
 
-![interface](interface.png)
+![interface](media/interface.png)
 
diff --git a/deepcave/__init__.py b/deepcave/__init__.py
index 16a396a9..7246f7a9 100644
--- a/deepcave/__init__.py
+++ b/deepcave/__init__.py
@@ -1,31 +1,50 @@
-import os
 import sys
-from deepcave.runs.recorder import Recorder
-from deepcave.runs.objective import Objective
 
+author = "René Sass and  Marius Lindauer"
+version = "0.2"
 
-version = "0.1"
-exec_file = sys.argv[0]
 
+_exec_file = sys.argv[0]
 
-if "server.py" in exec_file or "worker.py" in exec_file:
+if "server.py" in _exec_file or "worker.py" in _exec_file:
+    from deepcave.config import config
+    from deepcave.queue import Queue  # noqa
+    from deepcave.runs.handler import RunHandler  # noqa
+    from deepcave.runs.objective import Objective
+    from deepcave.runs.recorder import Recorder
+    from deepcave.server import get_app  # noqa
     from deepcave.utils.cache import Cache  # noqa
     from deepcave.utils.run_caches import RunCaches  # noqa
-    from deepcave.server import get_app  # noqa
-    from deepcave.queue import Queue  # noqa
-    from deepcave.config import CONFIG, META  # noqa
 
     app = get_app()
-    queue = Queue(CONFIG["REDIS_URL"])
+    queue = Queue(config.REDIS_ADDRESS, config.REDIS_PORT)
 
     # Meta cache
-    c = Cache(
-        filename=os.path.join(CONFIG["CACHE_DIR"], "meta.json"),
-        defaults=META)
+    c = Cache(filename=config.CACHE_DIR / "meta.json", defaults=config.META_DEFAULT)
 
     # Run caches
-    rc = RunCaches()
-
-    __all__ = ["version", "app", "queue", "c", "rc", "Recorder", "Objective"]
+    rc = RunCaches(config)
+
+    # Run Handler
+    run_handler = RunHandler(config, c, rc)
+
+    __all__ = [
+        "version",
+        "app",
+        "queue",
+        "c",
+        "rc",
+        "run_handler",
+        "config",
+        "Recorder",
+        "Objective",
+    ]
 else:
-    __all__ = ["version", "Recorder", "Objective"]
+
+    try:
+        from deepcave.runs.objective import Objective
+        from deepcave.runs.recorder import Recorder
+
+        __all__ = ["version", "Recorder", "Objective"]
+    except ModuleNotFoundError:
+        __all__ = ["version"]
diff --git a/deepcave/assets/custom.css b/deepcave/assets/custom.css
index 558098b9..867ab576 100644
--- a/deepcave/assets/custom.css
+++ b/deepcave/assets/custom.css
@@ -39,6 +39,15 @@ label {
 #sidebarMenu {
     background-color: rgb(243, 242, 241);
     border-right: 1px solid rgba(0, 0, 0, 0.1);
+    padding: 0px;
+}
+
+thead {
+    border-bottom: 2px solid lightgray;
+}
+
+.nav-link .far, .nav-link .fas {
+    margin-right: 0.5rem;
 }
 
 
diff --git a/deepcave/cli.py b/deepcave/cli.py
new file mode 100644
index 00000000..f0769d88
--- /dev/null
+++ b/deepcave/cli.py
@@ -0,0 +1,31 @@
+from typing import Any, List
+
+import subprocess
+
+from absl import app, flags
+
+FLAGS = flags.FLAGS
+flags.DEFINE_boolean("start", False, "Starts DeepCAVE.")
+flags.DEFINE_boolean("docker", False, "Uses docker images to start DeepCAVE.")
+flags.DEFINE_string("get_config", None, "Returns the value of a given config key.")
+
+
+def execute(args: List[Any]) -> None:
+    if (config_key := FLAGS.get_config) is not None:
+        from deepcave.config import config
+
+        print(getattr(config, config_key))
+        return
+
+    if FLAGS.start:
+        if FLAGS.docker:
+            # TODO: Implement docker execution
+            exit("The command is not supported yet.")
+
+            # subprocess.call('./start_docker.sh')
+        else:
+            subprocess.call("./start.sh")
+
+
+def main() -> None:
+    app.run(execute)
diff --git a/deepcave/config.py b/deepcave/config.py
index 31afa26c..8e014174 100644
--- a/deepcave/config.py
+++ b/deepcave/config.py
@@ -1,24 +1,75 @@
-import os
-# from appdirs import user_cache_dir
+from typing import Type
 
-root = os.getcwd()
-cache_dir = os.path.join(root, "cache")  # user_cache_dir("deepcave")
-working_dir = os.path.join(root, "examples", "logs", "DeepCAVE")
+from pathlib import Path
 
 
-# General information to start services
-CONFIG = {
-    'CACHE_DIR': cache_dir,
-    'REDIS_URL': "redis://localhost:6379",
-}
+class Config:
+    # General config
+    TITLE: str = "DeepCAVE"
 
-# Meta information which are used across the platform
-META = {
-    'matplotlib-mode': False,
-    'working_dir': working_dir,
-    'run_ids': {},  # {run_name: run_id}
-    'groups': {}  # {group_name: [run_name, ...]}
-}
+    # Cache dir
+    root = Path.cwd()
+    DEFAULT_WORKING_DIRECTORY = root / "examples" / "logs" / "DeepCAVE"
 
+    CACHE_DIR = root / "cache"
 
-__all__ = [CONFIG, META]
+    # Redis settings
+    REDIS_PORT = 6379
+    REDIS_ADDRESS = "redis://localhost"
+
+    # Dash settings (not used right now)
+    DASH_PORT = 8050
+    DASH_ADDRESS = "http://127.0.0.1"
+
+    # Default Meta information which are used across the platform
+    META_DEFAULT = {
+        "matplotlib-mode": False,
+        "working_dir": str(DEFAULT_WORKING_DIRECTORY),
+        "selected_runs": [],  # [run_name, ...]
+        "groups": {},  # {group_name: [run_name, ...]}
+    }
+
+    # Plugins
+    @property
+    def PLUGINS(self) -> dict[str, list[Type["Plugin"]]]:
+        """
+        Returns:
+        dictionary [category -> List[Plugins]]
+        Plugins are ordered
+        """
+        from deepcave.plugins.dynamic_plugin.ccube import CCube
+        from deepcave.plugins.dynamic_plugin.configurations import Configurations
+        from deepcave.plugins.dynamic_plugin.cost_over_time import CostOverTime
+        from deepcave.plugins.dynamic_plugin.overview import Overview
+        from deepcave.plugins.dynamic_plugin.parallel_coordinates import (
+            ParallelCoordinates,
+        )
+        from deepcave.plugins.static_plugin.fanova import fANOVA
+
+        plugins = {
+            "Summary": [
+                Overview(),
+                Configurations(),
+            ],
+            "Performance Analysis": [
+                CostOverTime(),
+                CCube(),
+                ParallelCoordinates(),
+            ],
+            "Hyperparameter Analysis": [
+                fANOVA(),
+            ],
+        }
+        return plugins
+
+    # Run Converter
+    @property
+    def AVAILABLE_CONVERTERS(self) -> list[Type["Run"]]:
+        from deepcave.runs.converters.bohb import BOHBRun
+        from deepcave.runs.converters.deepcave import DeepCAVERun
+        from deepcave.runs.converters.smac import SMACRun
+
+        return [DeepCAVERun, BOHBRun, SMACRun]
+
+
+config = Config()
diff --git a/deepcave/evaluators/epm/fanova_forest.py b/deepcave/evaluators/epm/fanova_forest.py
index a18ba2a6..8aa8f36e 100644
--- a/deepcave/evaluators/epm/fanova_forest.py
+++ b/deepcave/evaluators/epm/fanova_forest.py
@@ -1,50 +1,49 @@
-import pyrfr
-import pyrfr.regression as regression
-import pyrfr.util
-import typing
+from typing import Optional
+
 import itertools as it
 
 import numpy as np
+import pyrfr
+import pyrfr.regression as regression
+import pyrfr.util
 from smac.configspace import ConfigurationSpace
+
 from deepcave.evaluators.epm.forest import Forest
 
 
 class fANOVAForest(Forest):
     def __init__(
-            self,
-            configspace: ConfigurationSpace,
-            seed: int,
-            num_trees: int = 16,
-            bootstrapping=True,
-            points_per_tree=-1,
-            ratio_features: float = 7. / 10.,
-            min_samples_split=0,
-            min_samples_leaf=0,
-            max_depth=64,
-            cutoffs=(-np.inf, np.inf),
-
-            instance_features: typing.Optional[np.ndarray] = None,
-            pca_components: typing.Optional[int] = None):
-
-        super().__init__(
-            configspace,
-            seed,
-            instance_features,
-            pca_components
-        )
+        self,
+        configspace: ConfigurationSpace,
+        seed: int,
+        num_trees: int = 16,
+        bootstrapping=True,
+        points_per_tree=-1,
+        ratio_features: float = 7.0 / 10.0,
+        min_samples_split=0,
+        min_samples_leaf=0,
+        max_depth=64,
+        cutoffs=(-np.inf, np.inf),
+        instance_features: Optional[np.ndarray] = None,
+        pca_components: Optional[int] = None,
+    ):
+
+        super().__init__(configspace, seed, instance_features, pca_components)
 
         max_features = 0
         if ratio_features <= 1.0:
             max_features = max(1, int(len(self.types) * ratio_features))
 
-        self._set_model_options({
-            'num_trees': num_trees,
-            'do_bootstrapping': bootstrapping,
-            'tree_opts.max_features': max_features,
-            'tree_opts.min_samples_to_split': min_samples_split,
-            'tree_opts.min_samples_in_leaf': min_samples_leaf,
-            'tree_opts.max_depth': max_depth,
-        })
+        self._set_model_options(
+            {
+                "num_trees": num_trees,
+                "do_bootstrapping": bootstrapping,
+                "tree_opts.max_features": max_features,
+                "tree_opts.min_samples_to_split": min_samples_split,
+                "tree_opts.min_samples_in_leaf": min_samples_leaf,
+                "tree_opts.max_depth": max_depth,
+            }
+        )
 
         self.cutoffs = cutoffs
         self.points_per_tree = points_per_tree
@@ -56,7 +55,7 @@ def _get_model(self):
     def _train(self, X, Y):
         """
         Inputs:
-            `X`: Must be numerical encoded. 
+            `X`: Must be numerical encoded.
         """
 
         super()._train(X, Y)
@@ -86,8 +85,9 @@ def _train(self, X, Y):
                         sizes.append((self.bounds[i][0],))
                 else:
                     # add bounds to split values
-                    sv = np.array([self.bounds[i][0]] +
-                                  list(split_vals) + [self.bounds[i][1]])
+                    sv = np.array(
+                        [self.bounds[i][0]] + list(split_vals) + [self.bounds[i][1]]
+                    )
                     # compute midpoints and sizes
                     midpoints.append((1 / 2) * (sv[1:] + sv[:-1]))
                     sizes.append(sv[1:] - sv[:-1])
@@ -117,7 +117,7 @@ def _set_cutoffs(self, cutoffs=(-np.inf, np.inf), quantile=None):
         To properly do things like 'improvement over default' the
         fANOVA now supports cutoffs on the y values. These will exclude
         parts of the parameters space where the prediction is not within
-        the provided cutoffs. This is is specialization of 
+        the provided cutoffs. This is is specialization of
         "Generalized Functional ANOVA Diagnostics for High Dimensional
         Functions of Dependent Variables" by Hooker.
         """
@@ -158,13 +158,12 @@ def compute_marginals(self, dimensions, depth=1):
         self.V_U_individual[dimensions] = []
         self.V_U_total[dimensions] = []
 
-        if len(dimensions) > depth+1:
+        if len(dimensions) > depth + 1:
             return self.V_U_individual, self.V_U_total
 
         for tree_idx in range(len(self.all_midpoints)):
             # collect all the midpoints and corresponding sizes for that tree
-            midpoints = [self.all_midpoints[tree_idx][dim]
-                         for dim in dimensions]
+            midpoints = [self.all_midpoints[tree_idx][dim] for dim in dimensions]
             sizes = [self.all_sizes[tree_idx][dim] for dim in dimensions]
             stat = pyrfr.util.weighted_running_stats()
 
@@ -178,11 +177,11 @@ def compute_marginals(self, dimensions, depth=1):
                 sample[list(dimensions)] = list(m)
 
                 ls = self.model.marginal_prediction_stat_of_tree(
-                    tree_idx, sample.tolist())
+                    tree_idx, sample.tolist()
+                )
                 # self.logger.debug("%s, %s", (sample, ls.mean()))
                 if not np.isnan(ls.mean()):
-                    stat.push(ls.mean(), np.prod(np.array(s))
-                              * ls.sum_of_weights())
+                    stat.push(ls.mean(), np.prod(np.array(s)) * ls.sum_of_weights())
 
             # line 10 in algorithm 2
             # note that V_U^2 can be computed by var(\hat a)^2 - \sum_{subU} var(f_subU)^2
diff --git a/deepcave/evaluators/epm/forest.py b/deepcave/evaluators/epm/forest.py
index bc427b77..e397ffe3 100644
--- a/deepcave/evaluators/epm/forest.py
+++ b/deepcave/evaluators/epm/forest.py
@@ -1,20 +1,22 @@
 from abc import abstractmethod
-import pyrfr.regression as regression
-import typing
+
 import functools
 
-import copy
 import numpy as np
-
+import pyrfr.regression as regression
 from smac.configspace import ConfigurationSpace
-from smac.utils.constants import N_TREES, VERY_SMALL_NUMBER
-from smac.epm.rf_with_instances import RandomForestWithInstances as RFI
-from smac.epm.util_funcs import get_types
 from smac.epm.base_rf import BaseModel
+from smac.epm.util_funcs import get_types
 
 
 class Forest(BaseModel):
-    def __init__(self, configspace, seed=0, instance_features=None, pca_components=None):
+    def __init__(
+        self,
+        configspace: ConfigurationSpace,
+        seed=0,
+        instance_features=None,
+        pca_components=None,
+    ):
         # Set types and bounds automatically
         types, bounds = get_types(configspace, instance_features)
 
@@ -34,7 +36,7 @@ def __init__(self, configspace, seed=0, instance_features=None, pca_components=N
         self.rng = regression.default_random_engine(seed)
 
     @abstractmethod
-    def _get_model():
+    def _get_model(self):
         raise NotImplementedError()
 
     def _set_model_options(self, d):
@@ -43,16 +45,17 @@ def _set_model_options(self, d):
         def rgetattr(obj, attr, *args):
             def _getattr(obj, attr):
                 return getattr(obj, attr, *args)
-            return functools.reduce(_getattr, [obj] + attr.split('.'))
+
+            return functools.reduce(_getattr, [obj] + attr.split("."))
 
         def rsetattr(obj, attr, val):
-            pre, _, post = attr.rpartition('.')
+            pre, _, post = attr.rpartition(".")
             return setattr(rgetattr(obj, pre) if pre else obj, post, val)
 
         for k, v in d.items():
             rsetattr(self.model_options, k, v)
 
-    def _train(self, X: np.ndarray, y: np.ndarray) -> 'Forest':
+    def _train(self, X: np.ndarray, y: np.ndarray) -> "Forest":
         """Trains the random forest on X and y.
         Parameters
         ----------
@@ -77,7 +80,9 @@ def _train(self, X: np.ndarray, y: np.ndarray) -> 'Forest':
 
         return self
 
-    def _init_data_container(self, X: np.ndarray, y: np.ndarray) -> regression.default_data_container:
+    def _init_data_container(
+        self, X: np.ndarray, y: np.ndarray
+    ) -> regression.default_data_container:
         """Fills a pyrfr default data container, s.t. the forest knows
         categoricals and bounds for continous data
         Parameters
diff --git a/deepcave/evaluators/epm/random_forest.py b/deepcave/evaluators/epm/random_forest.py
new file mode 100644
index 00000000..ed56a1d0
--- /dev/null
+++ b/deepcave/evaluators/epm/random_forest.py
@@ -0,0 +1,97 @@
+from typing import Optional
+
+import numpy as np
+import pyrfr.regression as regression
+from smac.configspace import ConfigurationSpace
+from smac.utils.constants import N_TREES, VERY_SMALL_NUMBER
+
+from deepcave.evaluators.epm.forest import Forest
+
+
+class RandomForest(Forest):
+    """
+    Simple wrapper to calculate types and bounds automatically.
+    """
+
+    def __init__(
+        self,
+        configspace: ConfigurationSpace,
+        seed: int,
+        num_trees: int = N_TREES,
+        bootstrapping: bool = True,
+        points_per_tree: int = -1,
+        ratio_features: float = 5.0 / 6.0,
+        min_samples_split: int = 3,
+        min_samples_leaf: int = 3,
+        max_depth: int = 2**20,
+        eps_purity: float = 1e-8,
+        max_num_nodes: int = 2**20,
+    ):
+
+        super().__init__(configspace, seed)
+
+        max_features = 0
+        if ratio_features <= 1.0:
+            max_features = max(1, int(len(self.types) * ratio_features))
+
+        self._set_model_options(
+            {
+                "num_trees": num_trees,
+                "do_bootstrapping": bootstrapping,
+                "tree_opts.max_features": max_features,
+                "tree_opts.min_samples_to_split": min_samples_split,
+                "tree_opts.min_samples_in_leaf": min_samples_leaf,
+                "tree_opts.max_depth": max_depth,
+                "tree_opts.epsilon_purity": eps_purity,
+                "tree_opts.max_num_nodes": max_num_nodes,
+                "compute_law_of_total_variance": False,
+            }
+        )
+
+        self.points_per_tree = points_per_tree
+
+    def _get_model(self):
+        return regression.binary_rss_forest()
+
+    def _predict(
+        self, X: np.ndarray, cov_return_type: Optional[str] = "diagonal_cov"
+    ) -> tuple[np.ndarray, np.ndarray]:
+        """Predict means and variances for given X.
+        Parameters
+        ----------
+        X : np.ndarray of shape = [n_samples,
+                                   n_features (config + instance features)]
+        cov_return_type: Optional[str]
+            Specifies what to return along with the mean. Refer ``predict()`` for more information.
+        Returns
+        -------
+        means : np.ndarray of shape = [n_samples, 1]
+            Predictive mean
+        vars : np.ndarray  of shape = [n_samples, 1]
+            Predictive variance
+        """
+
+        if len(X.shape) != 2:
+            raise ValueError("Expected 2d array, got %dd array!" % len(X.shape))
+        if X.shape[1] != len(self.types):
+            raise ValueError(
+                "Rows in X should have %d entries but have %d!"
+                % (len(self.types), X.shape[1])
+            )
+        if cov_return_type != "diagonal_cov":
+            raise ValueError(
+                "'cov_return_type' can only take 'diagonal_cov' for this model"
+            )
+
+        X = self._impute_inactive(X)
+
+        means, vars_ = [], []
+        for row_X in X:
+            mean_, var = self.model.predict_mean_var(row_X)
+            means.append(mean_)
+            vars_.append(var)
+
+        means = np.array(means)
+        vars_ = np.array(vars_)
+
+        return means, vars_  # means.reshape((-1, 1)), vars_.reshape((-1, 1))
diff --git a/deepcave/evaluators/epm/random_forest_with_instances.py b/deepcave/evaluators/epm/random_forest_with_instances.py
index cad0306e..32decc1b 100644
--- a/deepcave/evaluators/epm/random_forest_with_instances.py
+++ b/deepcave/evaluators/epm/random_forest_with_instances.py
@@ -1,7 +1,7 @@
-import pyrfr.regression as regression
-import typing
+from typing import Optional
 
 import numpy as np
+import pyrfr.regression as regression
 from smac.configspace import ConfigurationSpace
 from smac.utils.constants import N_TREES, VERY_SMALL_NUMBER
 
@@ -14,44 +14,42 @@ class RandomForestWithInstances(Forest):
     """
 
     def __init__(
-            self,
-            configspace: ConfigurationSpace,
-            seed: int,
-            log_y: bool = False,
-            num_trees: int = N_TREES,
-            bootstrapping: bool = True,
-            points_per_tree: int = -1,
-            ratio_features: float = 5. / 6.,
-            min_samples_split: int = 3,
-            min_samples_leaf: int = 3,
-            max_depth: int = 2**20,
-            eps_purity: float = 1e-8,
-            max_num_nodes: int = 2**20,
-            instance_features: typing.Optional[np.ndarray] = None,
-            pca_components: typing.Optional[int] = None):
-
-        super().__init__(
-            configspace,
-            seed,
-            instance_features,
-            pca_components
-        )
+        self,
+        configspace: ConfigurationSpace,
+        seed: int,
+        log_y: bool = False,
+        num_trees: int = N_TREES,
+        bootstrapping: bool = True,
+        points_per_tree: int = -1,
+        ratio_features: float = 5.0 / 6.0,
+        min_samples_split: int = 3,
+        min_samples_leaf: int = 3,
+        max_depth: int = 2**20,
+        eps_purity: float = 1e-8,
+        max_num_nodes: int = 2**20,
+        instance_features: Optional[np.ndarray] = None,
+        pca_components: Optional[int] = None,
+    ):
+
+        super().__init__(configspace, seed, instance_features, pca_components)
 
         max_features = 0
         if ratio_features <= 1.0:
             max_features = max(1, int(len(self.types) * ratio_features))
 
-        self._set_model_options({
-            'num_trees': num_trees,
-            'do_bootstrapping': bootstrapping,
-            'tree_opts.max_features': max_features,
-            'tree_opts.min_samples_to_split': min_samples_split,
-            'tree_opts.min_samples_in_leaf': min_samples_leaf,
-            'tree_opts.max_depth': max_depth,
-            'tree_opts.epsilon_purity': eps_purity,
-            'tree_opts.max_num_nodes': max_num_nodes,
-            'compute_law_of_total_variance': False,
-        })
+        self._set_model_options(
+            {
+                "num_trees": num_trees,
+                "do_bootstrapping": bootstrapping,
+                "tree_opts.max_features": max_features,
+                "tree_opts.min_samples_to_split": min_samples_split,
+                "tree_opts.min_samples_in_leaf": min_samples_leaf,
+                "tree_opts.max_depth": max_depth,
+                "tree_opts.epsilon_purity": eps_purity,
+                "tree_opts.max_num_nodes": max_num_nodes,
+                "compute_law_of_total_variance": False,
+            }
+        )
 
         self.log_y = log_y
         self.points_per_tree = points_per_tree
@@ -59,15 +57,15 @@ def __init__(
     def _get_model(self):
         return regression.binary_rss_forest()
 
-    def _predict(self, X: np.ndarray,
-                 cov_return_type: typing.Optional[str] = 'diagonal_cov') \
-            -> typing.Tuple[np.ndarray, np.ndarray]:
+    def _predict(
+        self, X: np.ndarray, cov_return_type: Optional[str] = "diagonal_cov"
+    ) -> tuple[np.ndarray, np.ndarray]:
         """Predict means and variances for given X.
         Parameters
         ----------
         X : np.ndarray of shape = [n_samples,
                                    n_features (config + instance features)]
-        cov_return_type: typing.Optional[str]
+        cov_return_type: Optional[str]
             Specifies what to return along with the mean. Refer ``predict()`` for more information.
         Returns
         -------
@@ -77,14 +75,16 @@ def _predict(self, X: np.ndarray,
             Predictive variance
         """
         if len(X.shape) != 2:
-            raise ValueError(
-                'Expected 2d array, got %dd array!' % len(X.shape))
+            raise ValueError("Expected 2d array, got %dd array!" % len(X.shape))
         if X.shape[1] != len(self.types):
-            raise ValueError('Rows in X should have %d entries but have %d!' % (
-                len(self.types), X.shape[1]))
-        if cov_return_type != 'diagonal_cov':
             raise ValueError(
-                "'cov_return_type' can only take 'diagonal_cov' for this model")
+                "Rows in X should have %d entries but have %d!"
+                % (len(self.types), X.shape[1])
+            )
+        if cov_return_type != "diagonal_cov":
+            raise ValueError(
+                "'cov_return_type' can only take 'diagonal_cov' for this model"
+            )
 
         X = self._impute_inactive(X)
 
@@ -100,15 +100,18 @@ def _predict(self, X: np.ndarray,
                 third_dimension = max(max_num_leaf_data, third_dimension)
 
             # Transform list of 2d arrays into a 3d array
-            preds_as_array = np.zeros(
-                (X.shape[0], self.forest_options.num_trees, third_dimension)) * np.NaN
+            preds_as_array = (
+                np.zeros((X.shape[0], self.forest_options.num_trees, third_dimension))
+                * np.NaN
+            )
             for i, preds_per_tree in enumerate(all_preds):
                 for j, pred in enumerate(preds_per_tree):
-                    preds_as_array[i, j, :len(pred)] = pred
+                    preds_as_array[i, j, : len(pred)] = pred
 
             # Do all necessary computation with vectorized functions
-            preds_as_array = np.log(np.nanmean(
-                np.exp(preds_as_array), axis=2) + VERY_SMALL_NUMBER)
+            preds_as_array = np.log(
+                np.nanmean(np.exp(preds_as_array), axis=2) + VERY_SMALL_NUMBER
+            )
 
             # Compute the mean and the variance across the different trees
             means = preds_as_array.mean(axis=1)
@@ -125,7 +128,9 @@ def _predict(self, X: np.ndarray,
 
         return means.reshape((-1, 1)), vars_.reshape((-1, 1))
 
-    def predict_marginalized_over_instances(self, X: np.ndarray) -> typing.Tuple[np.ndarray, np.ndarray]:
+    def predict_marginalized_over_instances(
+        self, X: np.ndarray
+    ) -> tuple[np.ndarray, np.ndarray]:
         """Predict mean and variance marginalized over all instances.
         Returns the predictive mean and variance marginalised over all
         instances for a set of configurations.
@@ -148,8 +153,7 @@ def predict_marginalized_over_instances(self, X: np.ndarray) -> typing.Tuple[np.
             Predictive variance
         """
 
-        if self.instance_features is None or \
-                len(self.instance_features) == 0:
+        if self.instance_features is None or len(self.instance_features) == 0:
             mean_, var = self.predict(X)
             assert var is not None  # please mypy
 
@@ -158,12 +162,12 @@ def predict_marginalized_over_instances(self, X: np.ndarray) -> typing.Tuple[np.
             return mean_, var
 
         if len(X.shape) != 2:
-            raise ValueError(
-                'Expected 2d array, got %dd array!' % len(X.shape))
+            raise ValueError("Expected 2d array, got %dd array!" % len(X.shape))
         if X.shape[1] != len(self.bounds):
-            raise ValueError('Rows in X should have %d entries but have %d!' %
-                             (len(self.bounds),
-                              X.shape[1]))
+            raise ValueError(
+                "Rows in X should have %d entries but have %d!"
+                % (len(self.bounds), X.shape[1])
+            )
 
         X = self._impute_inactive(X)
 
@@ -173,7 +177,7 @@ def predict_marginalized_over_instances(self, X: np.ndarray) -> typing.Tuple[np.
 
             # marginalize over instances
             # 1. get all leaf values for each tree
-            # type: typing.List[typing.List[float]]
+            # type: list[list[float]]
             preds_trees = [[] for i in range(self.forest_options.num_trees)]
 
             for feat in self.instance_features:
@@ -185,8 +189,9 @@ def predict_marginalized_over_instances(self, X: np.ndarray) -> typing.Tuple[np.
             # 2. average in each tree
             if self.log_y:
                 for tree_id in range(self.forest_options.num_trees):
-                    dat_[i, tree_id] = \
-                        np.log(np.exp(np.array(preds_trees[tree_id])).mean())
+                    dat_[i, tree_id] = np.log(
+                        np.exp(np.array(preds_trees[tree_id])).mean()
+                    )
             else:
                 for tree_id in range(self.forest_options.num_trees):
                     dat_[i, tree_id] = np.array(preds_trees[tree_id]).mean()
diff --git a/deepcave/evaluators/fanova.py b/deepcave/evaluators/fanova.py
index ae7a2978..1f8a09fd 100644
--- a/deepcave/evaluators/fanova.py
+++ b/deepcave/evaluators/fanova.py
@@ -1,27 +1,33 @@
+from typing import Optional, Union
 
 import itertools as it
-import typing
 from collections import OrderedDict
+
 import numpy as np
+import pandas as pd
+from ConfigSpace import ConfigurationSpace
 
 
 class fANOVA:
-    def __init__(self,
-                 X, Y,
-                 configspace=None,
-                 seed=0,
-                 num_trees=16,
-                 bootstrapping=True,
-                 points_per_tree=-1,
-                 ratio_features: float = 7. / 10.,
-                 min_samples_split=0,
-                 min_samples_leaf=0,
-                 max_depth=64,
-                 cutoffs=(-np.inf, np.inf),
-                 instance_features: typing.Optional[np.ndarray] = None,
-                 pca_components: typing.Optional[int] = None):
+    def __init__(
+        self,
+        X: Union[pd.DataFrame, np.ndarray],
+        Y,
+        configspace: ConfigurationSpace,
+        seed=0,
+        num_trees=16,
+        bootstrapping=True,
+        points_per_tree=-1,
+        ratio_features: float = 7 / 10,
+        min_samples_split=0,
+        min_samples_leaf=0,
+        max_depth=64,
+        cutoffs=(-np.inf, np.inf),
+        instance_features: Optional[np.ndarray] = None,
+        pca_components: Optional[int] = None,
+    ):
         """
-        Calculate and provide midpoints and sizes from the forest's 
+        Calculate and provide midpoints and sizes from the forest's
         split values in order to get the marginals
 
         Parameters
@@ -30,20 +36,20 @@ def __init__(self,
 
         Y: vector with the response values (numerically encoded)
 
-        config_space : ConfigSpace instantiation
+        configspace : ConfigSpace instantiation
 
         num_trees: number of trees in the forest to be fit
 
         seed: seed for the forests randomness
 
-        bootstrapping: whether or not to bootstrap the data for each tree
+        bootstrapping: whether to bootstrap the data for each tree or not
 
-        points_per_tree: number of points used for each tree 
+        points_per_tree: number of points used for each tree
                         (only subsampling if bootstrapping is false)
 
-        max_features: number of features to be used at each split, default is 70%
+        ratio_features: number of features to be used at each split, default is 70%
 
-        min_samples_split: minimum number of samples required to attempt to split 
+        min_samples_split: minimum number of samples required to attempt to split
 
         min_samples_leaf: minimum number of samples required in a leaf
 
@@ -79,13 +85,21 @@ def __init__(self,
 
         self.forest.train(X, Y)
 
-    def quantify_importance(self, dims, depth=1, sorted=True):
+    def quantify_importance(
+        self, dims, depth=1, sort=True
+    ) -> dict[tuple, tuple[float, float, float, float]]:
         """
         Inputs:
-            `depth`: How often dims should be combinated.
+            `depth`: How often dims should be combined.
 
         Returns:
             ordered dict on total importance
+            Dict[Tuple[dim_names] -> (
+                                mean_fractions_individual,
+                                mean_fractions_total,
+                                std_fractions_individual,
+                                std_fractions_total
+                            )]
         """
 
         if type(dims[0]) == str:
@@ -97,8 +111,7 @@ def quantify_importance(self, dims, depth=1, sorted=True):
         else:
             dimensions = dims
 
-        vu_individual, vu_total = self.forest.compute_marginals(
-            dimensions, depth)
+        vu_individual, vu_total = self.forest.compute_marginals(dimensions, depth)
 
         importance_dict = {}
 
@@ -110,23 +123,31 @@ def quantify_importance(self, dims, depth=1, sorted=True):
                 if type(dims[0]) == str:
                     dim_names = []
                     for j, dim in enumerate(sub_dims):
-                        dim_names.append(
-                            self.cs.get_hyperparameter_by_idx(dim))
+                        dim_names.append(self.cs.get_hyperparameter_by_idx(dim))
                     dim_names = tuple(dim_names)
                     importance_dict[dim_names] = {}
                 else:
                     importance_dict[sub_dims] = {}
+
                 # clean here to catch zero variance in a trees
                 non_zero_idx = np.nonzero(
-                    [self.forest.trees_total_variance[t] for t in range(self.num_trees)])
+                    [self.forest.trees_total_variance[t] for t in range(self.num_trees)]
+                )
                 if len(non_zero_idx[0]) == 0:
-                    raise RuntimeError(
-                        'Encountered zero total variance in all trees.')
+                    raise RuntimeError("Encountered zero total variance in all trees.")
 
-                fractions_total = np.array([vu_total[sub_dims][t] / self.forest.trees_total_variance[t]
-                                            for t in non_zero_idx[0]])
-                fractions_individual = np.array([vu_individual[sub_dims][t] / self.forest.trees_total_variance[t]
-                                                 for t in non_zero_idx[0]])
+                fractions_total = np.array(
+                    [
+                        vu_total[sub_dims][t] / self.forest.trees_total_variance[t]
+                        for t in non_zero_idx[0]
+                    ]
+                )
+                fractions_individual = np.array(
+                    [
+                        vu_individual[sub_dims][t] / self.forest.trees_total_variance[t]
+                        for t in non_zero_idx[0]
+                    ]
+                )
 
                 if type(dims[0]) == str:
                     sub_dims = dim_names
@@ -138,9 +159,11 @@ def quantify_importance(self, dims, depth=1, sorted=True):
                     np.std(fractions_total),
                 )
 
-        if sorted:
-            sorted_importance_dict = {k: v for k, v in sorted(
-                importance_dict.items(), key=lambda item: item[1][1])}
+        if sort:
+            sorted_importance_dict = {
+                k: v
+                for k, v in sorted(importance_dict.items(), key=lambda item: item[1][1])
+            }
 
             return sorted_importance_dict
 
@@ -153,14 +176,14 @@ def marginal_mean_variance_for_values(self, dimlist, values_to_predict):
         Parameters
         ----------
         dimlist: list
-                Contains the indices of ConfigSpace for the selected parameters 
-                (starts with 0) 
+                Contains the indices of ConfigSpace for the selected parameters
+                (starts with 0)
         values_to_predict: list
                 Contains the values to be predicted
 
         Returns
         -------
-        tuple 
+        tuple
             marginal mean prediction and corresponding variance estimate
         """
         sample = np.full(self.n_dims, np.nan, dtype=np.float)
@@ -182,7 +205,7 @@ def get_most_important_pairwise_marginals(self, params=None, n=10):
 
         Returns
         -------
-        list: 
+        list:
              Contains the n most important pairwise marginals
         """
         self.tot_imp_dict = OrderedDict()
@@ -204,14 +227,11 @@ def get_most_important_pairwise_marginals(self, params=None, n=10):
             n = len(list(pairs))
         for combi in pairs:
             pairwise_marginal_performance = self.quantify_importance(combi)
-            tot_imp = pairwise_marginal_performance[combi]['individual importance']
-            combi_names = [self.cs_params[combi[0]].name,
-                           self.cs_params[combi[1]].name]
-            pairwise_marginals.append(
-                (tot_imp, combi_names[0], combi_names[1]))
+            tot_imp = pairwise_marginal_performance[combi]["individual importance"]
+            combi_names = [self.cs_params[combi[0]].name, self.cs_params[combi[1]].name]
+            pairwise_marginals.append((tot_imp, combi_names[0], combi_names[1]))
 
-        pairwise_marginal_performance = sorted(
-            pairwise_marginals, reverse=True)
+        pairwise_marginal_performance = sorted(pairwise_marginals, reverse=True)
 
         for marginal, p1, p2 in pairwise_marginal_performance[:n]:
             self.tot_imp_dict[(p1, p2)] = marginal
@@ -229,14 +249,16 @@ def get_triple_marginals(self, params=None):
 
         Returns
         -------
-        list: 
+        list:
              Contains most important triple marginals
         """
         self.tot_imp_dict = OrderedDict()
         triple_marginals = []
         if len(params) < 3:
             raise RuntimeError(
-                'Number of parameters have to be greater than %i. At least 3 parameters needed' % len(params))
+                "Number of parameters have to be greater than %i. At least 3 parameters needed"
+                % len(params)
+            )
         if type(params[0]) == str:
             idx = []
             for i, param in enumerate(params):
@@ -249,16 +271,21 @@ def get_triple_marginals(self, params=None):
         triplets = [x for x in it.combinations(dimensions, 3)]
         for combi in triplets:
             triple_marginal_performance = self.quantify_importance(combi)
-            tot_imp = triple_marginal_performance[combi]['individual importance']
-            combi_names = [self.cs_params[combi[0]].name,
-                           self.cs_params[combi[1]].name, self.cs_params[combi[2]].name]
+            tot_imp = triple_marginal_performance[combi]["individual importance"]
+            combi_names = [
+                self.cs_params[combi[0]].name,
+                self.cs_params[combi[1]].name,
+                self.cs_params[combi[2]].name,
+            ]
             triple_marginals.append(
-                (tot_imp, combi_names[0], combi_names[1], combi_names[2]))
+                (tot_imp, combi_names[0], combi_names[1], combi_names[2])
+            )
 
         triple_marginal_performance = sorted(triple_marginals, reverse=True)
         if params:
-            triple_marginal_performance = triple_marginal_performance[:len(
-                list(triplets))]
+            triple_marginal_performance = triple_marginal_performance[
+                : len(list(triplets))
+            ]
 
         for marginal, p1, p2, p3 in triple_marginal_performance:
             self.tot_imp_dict[(p1, p2, p3)] = marginal
@@ -268,25 +295,33 @@ def get_triple_marginals(self, params=None):
 
 if __name__ == "__main__":
     import sys
-    sys.path.insert(0, '../../')
 
-    import numpy as np
-    import ConfigSpace
+    sys.path.insert(0, "../../")
 
+    import ConfigSpace
     import ConfigSpace as CS
     import ConfigSpace.hyperparameters as CSH
-    from ConfigSpace.hyperparameters import CategoricalHyperparameter, Constant, UniformFloatHyperparameter, UniformIntegerHyperparameter
+    import numpy as np
+    from ConfigSpace.hyperparameters import (
+        CategoricalHyperparameter,
+        Constant,
+        UniformFloatHyperparameter,
+        UniformIntegerHyperparameter,
+    )
 
     cs = CS.ConfigurationSpace(seed=1234)
 
-    alpha = CSH.UniformFloatHyperparameter(name='alpha', lower=0, upper=1)
-    beta = CSH.UniformFloatHyperparameter(name='beta', lower=0, upper=1)
-    gamma = CSH.UniformFloatHyperparameter(name='gamma', lower=0, upper=1)
-    gamma1 = CSH.UniformFloatHyperparameter(name='gamma1', lower=0, upper=1)
-    gamma2 = CSH.UniformFloatHyperparameter(name='gamma2', lower=0, upper=1)
-    gamma3 = CSH.UniformFloatHyperparameter(name='gamma3', lower=0, upper=1)
+    alpha = CSH.UniformFloatHyperparameter(name="alpha", lower=0, upper=1)
+    beta = CSH.UniformFloatHyperparameter(name="beta", lower=0, upper=1)
+    gamma = CSH.UniformFloatHyperparameter(name="gamma", lower=0, upper=1)
+    gamma1 = CSH.UniformFloatHyperparameter(name="gamma1", lower=0, upper=1)
+    gamma2 = CSH.UniformFloatHyperparameter(name="gamma2", lower=0, upper=1)
+    gamma3 = CSH.UniformFloatHyperparameter(name="gamma3", lower=0, upper=1)
+
+    # Constants do not work
+    # gamma = CSH.Constant(name='gamma', value=1)
 
-    cs.add_hyperparameters([alpha, beta, gamma, gamma1, gamma2, gamma3])
+    cs.add_hyperparameters([alpha, beta, gamma])
 
     X = []
     Y = []
@@ -313,7 +348,9 @@ def get_triple_marginals(self, params=None):
                 conditional[idx] = True
                 if isinstance(hp, CategoricalHyperparameter):
                     impute_values[idx] = len(hp.choices)
-                elif isinstance(hp, (UniformFloatHyperparameter, UniformIntegerHyperparameter)):
+                elif isinstance(
+                    hp, (UniformFloatHyperparameter, UniformIntegerHyperparameter)
+                ):
                     impute_values[idx] = -1
                 elif isinstance(hp, Constant):
                     impute_values[idx] = 1
@@ -324,10 +361,10 @@ def get_triple_marginals(self, params=None):
             nonfinite_mask = ~np.isfinite(X[:, idx])
             X[nonfinite_mask, idx] = impute_values[idx]
 
-    #f = fANOVA(X, Y, cs)
-    #imp = f.quantify_importance(cs.get_hyperparameter_names()[:3], depth=1)
+    # f = fANOVA(X, Y, cs)
+    # imp = f.quantify_importance(cs.get_hyperparameter_names()[:3], depth=1)
     # print(imp)
 
     f = fANOVA(X, Y, cs)
-    imp = f.quantify_importance(cs.get_hyperparameter_names(), depth=1)
+    imp = f.quantify_importance(cs.get_hyperparameter_names(), depth=1, sorted=False)
     print(imp)
diff --git a/deepcave/evaluators/ice.py b/deepcave/evaluators/ice.py
index fb5f9b08..6ae3d005 100644
--- a/deepcave/evaluators/ice.py
+++ b/deepcave/evaluators/ice.py
@@ -1,42 +1,55 @@
+from typing import Any, Optional
+
 import numpy as np
-import pickle
-from sklearn.ensemble import RandomForestRegressor
+from ConfigSpace import ConfigurationSpace
+
+from deepcave.evaluators.epm.random_forest import RandomForest
 
 
 class ICE:
-    def __init__(self, data={}):
+    def __init__(self, data: Optional[dict[int, Any]] = None):
         self.model = None
 
         # Make sure to have int keys
         self.data = {}
 
-        for k, (X, Y) in data.items():
-
-            X = np.array(X)
-            Y = np.array(Y)
+        if data is not None:
+            for k, (X, Y_mean, Y_var) in data.items():
+                X = np.array(X)
+                Y_mean = np.array(Y_mean)
+                Y_var = np.array(Y_var)
 
-            self.data[int(k)] = (X, Y)
+                self.data[int(k)] = (X, Y_mean, Y_var)
 
-    def get_data(self):
+    def get_data(self) -> dict[int, Any]:
         return self.data
 
-    def fit(self, configspace, X, Y, seed=0):
-        """
-        Args:
-            s (int): The id of the requested hyperparameter.
-        """
-
+    def fit(
+        self, configspace: ConfigurationSpace, X: np.ndarray, Y: np.ndarray, seed=0
+    ):
         # Train random forest here
         if self.model is None:
-            self.model = RandomForestRegressor(random_state=seed)
-            self.model.fit(X, Y)
+            self.model = RandomForest(
+                configspace=configspace,
+                seed=seed,
+                # num_trees=num_trees,
+                bootstrapping=False,
+                # points_per_tree=points_per_tree,
+                # ratio_features=ratio_features,
+                # min_samples_split=min_samples_split,
+                # min_samples_leaf=min_samples_leaf,
+                # max_depth=max_depth,
+                # cutoffs=cutoffs,
+            )
+            self.model.train(X, Y)
 
         for hp_name in configspace.get_hyperparameter_names():
             s = configspace.get_idx_by_hyperparameter_name(hp_name)
 
             shape = X.shape
             X_ice = np.zeros((shape[0], *shape))
-            y_ice = np.zeros((shape[0], shape[0]))
+            y_ice_mean = np.zeros((shape[0], shape[0]))
+            y_ice_var = np.zeros((shape[0], shape[0]))
 
             # Iterate over data points
             for i, _ in enumerate(X):
@@ -49,15 +62,32 @@ def fit(self, configspace, X, Y, seed=0):
                 X_ice[i] = X_copy
 
                 # Then we do a prediction with the new data
-                y_ice[i] = self.model.predict(X_copy)
+                # print(self.model.__dict__)
+                mean, var = self.model.predict(X_copy)
+                var = var.reshape((-1,))
+                mean = mean.reshape((-1,))
+
+                y_ice_mean[i] = mean
+                y_ice_var[i] = var
 
-            self.data[int(s)] = (X_ice, y_ice)
+            self.data[int(s)] = (X_ice, y_ice_mean, y_ice_var)
 
-    def get_ice_data(self, s, centered=False):
+    def get_ice_data(
+        self, s: int, centered=False, variance_based=False
+    ) -> tuple[list[float], list[float]]:
+        """
+        Args:
+            s (int): The id of the requested hyperparameter.
+        """
         if s not in self.data:
             return [], []
 
-        (X_ice, y_ice) = self.data[s]
+        (X_ice, y_ice_mean, y_ice_var) = self.data[s]
+
+        if variance_based:
+            y_ice = y_ice_var
+        else:
+            y_ice = y_ice_mean
 
         all_x = []
         all_y = []
@@ -81,11 +111,22 @@ def get_ice_data(self, s, centered=False):
 
         return all_x, all_y
 
-    def get_pdp_data(self, s):
+    def get_pdp_data(
+        self, s: int, variance_based=False
+    ) -> tuple[list[float], list[float], list[float]]:
+        """
+        Args:
+            s (int): The id of the requested hyperparameter.
+        """
         if s not in self.data:
-            return [], []
+            return [], [], []
+
+        (X_ice, y_ice_mean, y_ice_var) = self.data[s]
 
-        (X_ice, y_ice) = self.data[s]
+        if variance_based:
+            y_ice = y_ice_var
+        else:
+            y_ice = y_ice_mean
 
         n = y_ice.shape[0]
 
@@ -93,18 +134,22 @@ def get_pdp_data(self, s):
         x = X_ice[:, 0, s]
 
         y = []
+        y_err = []
+
         # Simply take all values and mean them
         for i in range(n):
             m = np.mean(y_ice[i, :])
             y.append(m)
 
+            std = np.std(y_ice[i, :])
+            y_err.append(std)
+
         y = np.array(y)
+        y_err = np.array(y_err)
 
         idx = np.argsort(x)
         x = x[idx]
         y = y[idx]
+        y_err = y_err[idx]
 
-        # Let's calculate uncertainties here
-        m_s = np.mean(y)
-
-        return x, y
+        return x, y, y_err
diff --git a/examples/__init__.py b/deepcave/evaluators/pending/ablation.py
similarity index 100%
rename from examples/__init__.py
rename to deepcave/evaluators/pending/ablation.py
diff --git a/deepcave/evaluators/pending/base_evaluator.py b/deepcave/evaluators/pending/base_evaluator.py
new file mode 100644
index 00000000..7dcd4135
--- /dev/null
+++ b/deepcave/evaluators/pending/base_evaluator.py
@@ -0,0 +1,99 @@
+class AbstractEvaluator(object):
+    """
+    Abstract implementation of Importance evaluator
+    """
+    def __init__(self, scenario: Scenario,
+                 cs: ConfigurationSpace,
+                 model: RandomForestWithInstances,
+                 to_evaluate: int, rng,
+                 verbose: bool=True,
+                 **kwargs):
+        self._logger = None
+        self.scenario = scenario
+        self.cs = cs
+        self.model = model  # SMAC model
+        self.rng = rng
+        self.verbose = verbose
+
+        if self.model is not None:
+            if 'X' in kwargs and 'y' in kwargs:
+                self._train_model(kwargs['X'], kwargs['y'], **kwargs)
+            if 'features' in kwargs:
+                self.features = kwargs['features']
+            else:
+                self.features = self.model.instance_features
+
+            self.X = self.model.X
+            self.y = self.model.y
+            self.types = self.model.types
+            self.bounds = self.model.bounds
+        self._to_eval = to_evaluate
+        if to_evaluate <= 0:
+            self.to_evaluate = len(self.cs.get_hyperparameters())
+        elif to_evaluate >= len(self.cs.get_hyperparameters()):
+            self.to_evaluate = len(self.cs.get_hyperparameters())
+        else:
+            self.to_evaluate = to_evaluate  # num of parameters to evaluate
+
+        self.evaluated_parameter_importance = OrderedDict()
+        self.name = 'Base'
+
+        self.IMPORTANCE_THRESHOLD = 0.05
+        self.AXIS_FONT = {'family': 'monospace'}
+        self.LABEL_FONT = {'family': 'sans-serif'}
+        self.LINE_FONT = {'lw': 4,
+                          'color': (0.125, 0.125, 0.125)}
+        self.area_color = (0.25, 0.25, 0.45)
+        self.unimportant_area_color = (0.125, 0.125, 0.225)
+        self.MAX_PARAMS_TO_PLOT = 15
+
+    @abc.abstractclassmethod
+    def run(self) -> OrderedDict:
+        raise NotImplementedError
+
+    @abc.abstractclassmethod
+    def plot_result(self, name=None):
+        raise NotImplementedError
+
+    def _train_model(self, X, y, **kwargs):
+        self.model.train(X, y, **kwargs)
+
+    def __str__(self):
+        tmp = 'Parameter Importance Evaluation Method %s\n' % self.name
+        tmp += '{:^15s}: {:<8s}\n'.format('Parameter', 'Value')
+        for key in self.evaluated_parameter_importance:
+            value = self.evaluated_parameter_importance[key]
+            tmp += '{:>15s}: {:<3.4f}\n'.format(key, value)
+        return tmp
+
+    @property
+    def logger(self):
+        return self._logger
+
+    @logger.setter
+    def logger(self, value):
+        self._logger = logging.getLogger(value)
+
+    def _refit_model(self, types, bounds, X, y):
+        """
+        Easily allows for refitting of the model.
+        Parameters
+        ----------
+        types: list
+            SMAC EPM types
+        X:ndarray
+            X matrix
+        y:ndarray
+            corresponding y vector
+        """
+        # We need to fake config-space bypass imputation of inactive values in random forest implementation
+        fake_cs = ConfigurationSpace(name="fake-cs-for-configurator-footprint")
+        # We need to add fake hyperparameters
+        fake_cs.add_hyperparameters([
+            UniformFloatHyperparameter('fake-%s' % i, lower=0., upper=100000., default_value=0.,
+                                       log=False) for i in range(len(types))
+        ])
+
+        self.model = RandomForestWithInstances(fake_cs, types, bounds, seed=12345, do_bootstrapping=True)
+        self.model.rf_opts.compute_oob_error = True
+        self.model.train(X, y)
\ No newline at end of file
diff --git a/deepcave/evaluators/pending/importance.py b/deepcave/evaluators/pending/importance.py
new file mode 100644
index 00000000..ca3c23d7
--- /dev/null
+++ b/deepcave/evaluators/pending/importance.py
@@ -0,0 +1,522 @@
+from typing import Dict, List, Tuple, Union
+
+import glob
+import json
+import logging
+import os
+import sys
+from collections import OrderedDict
+
+import numpy as np
+from pimp.configspace import (
+    CategoricalHyperparameter,
+    Configuration,
+    impute_inactive_values,
+)
+from pimp.epm.base_epm import RandomForestWithInstances
+from pimp.epm.unlogged_epar_x_rfwi import UnloggedEPARXrfi
+from pimp.epm.unlogged_rfwi import Unloggedrfwi
+from pimp.evaluator.ablation import Ablation
+from pimp.evaluator.fanova import fANOVA
+from pimp.evaluator.forward_selection import AbstractEvaluator, ForwardSelector
+from pimp.evaluator.influence_models import InfluenceModel
+from pimp.evaluator.local_parameter_importance import LPI
+from pimp.utils import RunHistory, RunHistory2EPM4Cost, RunHistory2EPM4LogCost, Scenario
+from smac.epm.rfr_imputator import RFRImputator
+from smac.epm.util_funcs import get_types
+from smac.tae import StatusType
+from smac.utils.io.traj_logging import TrajLogger
+from tqdm import tqdm
+
+__author__ = "Andre Biedenkapp"
+__copyright__ = "Copyright 2016, ML4AAD"
+__license__ = "3-clause BSD"
+__maintainer__ = "Andre Biedenkapp"
+__email__ = "biedenka@cs.uni-freiburg.de"
+
+
+class Importance(object):
+    def __init__(self, scenario_file: Union[None, str] = None, scenario: Union[None, Scenario] = None,
+                 runhistory_file: Union[str, None] = None, runhistory: Union[None, RunHistory] = None,
+                 traj_file: Union[None, List[str]] = None, incumbent: Union[None, Configuration] = None,
+                 seed: int = 12345, parameters_to_evaluate: int = -1, margin: Union[None, float] = None,
+                 save_folder: str = 'PIMP', impute_censored: bool = False, max_sample_size: int = -1,
+                 fANOVA_cut_at_default=False, fANOVA_pairwise=True, forwardsel_feat_imp=False,
+                 incn_quant_var=True, preprocess=False, forwardsel_cv=False, verbose: bool=True):
+        """
+        Importance Object. Handles the construction of the data and training of the model. Easy interface to the
+        different evaluators.
+        :param scenario_file: File to load the scenario from, if scenario is None.
+        :param scenario: Scenario Object to use if scenario_file is None
+        :param runhistory_file: File to load the runhistory from if runhistory is None.
+        :param runhistory: Runhistory Object to use if runhistory_file is None.
+        :param traj_file: File to load the trajectory from. If this is None but runhistory_file was specified,
+               the trajectory will be read from the same directory as the runhistory. (Will be ignored if incumbent is
+               set)
+        :param incumbent: Configuration Object to use.
+        :param seed: Seed used for the numpy random generator.
+        :param parameters_to_evaluate: int that specifies how many parameters have to be evaluated.
+               If set to -1 all parameters will be evaluated.
+        :param margin: float used in conjunction with influence models. Is the minimal improvement to accept a
+                       parameter as important.
+        :param save_folder: Folder name to save the output to
+        :param impute_censored: boolean that specifies if censored data should be imputed. If not, censored data are
+               ignored.
+        :param verbose: Toggle output to stdout (not logging, but tqdm-progress bars)
+        """
+        self.logger = logging.getLogger("pimp.Importance")
+        self.rng = np.random.RandomState(seed)
+        self._parameters_to_evaluate = parameters_to_evaluate
+        self._evaluator = None
+        self.margin = margin
+        self.threshold = None
+        self.seed = seed
+        self.impute = impute_censored
+        self.cut_def_fan = fANOVA_cut_at_default
+        self.pairiwse_fANOVA = fANOVA_pairwise
+        self.forwardsel_feat_imp = forwardsel_feat_imp
+        self.incn_quant_var = incn_quant_var
+        self.preprocess = preprocess
+        self._preprocessed = False
+        self.X_fanova = None
+        self.y_fanova = None
+        self.forwardsel_cv = forwardsel_cv
+        self.verbose = verbose
+
+        self.evaluators = []
+
+        self._setup_scenario(scenario, scenario_file, save_folder)
+        self._load_runhist(runhistory, runhistory_file)
+        self._setup_model()
+        self.best_dir = None
+        self._load_incumbent(traj_file, runhistory_file, incumbent)
+        self.logger.info('Best incumbent found in %s' % self.best_dir)
+        if 0 < max_sample_size < len(self.X):
+            self.logger.warning('Reducing the amount of datapoints!')
+            if self.best_dir:
+                self.logger.warning('Only using the runhistory that contains the incumbent!')
+                self.logger.info('Loading from %s' % self.best_dir)
+                self._load_runhist(None, os.path.join(self.best_dir, '*history*'))
+                self._convert_data(fit=False)
+                self._load_incumbent(glob.glob(os.path.join(self.best_dir, '*traj_aclib2*'), recursive=True)[0], None,
+                                     incumbent)
+                if max_sample_size < len(self.X):
+                    self.logger.warning('Also downsampling as requested!')
+                    idx = list(range(len(self.X)))
+                    np.random.shuffle(idx)
+                    idx = idx[:max_sample_size]
+                    self.X = self.X[idx]
+                    self.y = self.y[idx]
+            else:
+                self.logger.warning('Downsampling as requested!')
+                idx = list(range(len(self.X)))
+                np.random.shuffle(idx)
+                idx = idx[:max_sample_size]
+                self.X = self.X[idx]
+                self.y = self.y[idx]
+            self.logger.info('Remaining %d datapoints' % len(self.X))
+            self.model.train(self.X, self.y)
+
+    def _preprocess(self, runhistory):
+        """
+        Method to marginalize over instances such that fANOVA can determine the parameter importance without
+        having to deal with instance features.
+        :param runhistory: RunHistory that knows all configurations that were run. For all these configurations
+                           we have to marginalize away the instance features with which fANOVA will make it's
+                           predictions
+        """
+        self.logger.info('PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING')
+        self.logger.info('Marginalizing away all instances!')
+        configs = runhistory.get_all_configs()
+        X_non_hyper, X_prime, y_prime = [], [], []
+        for c_id, config in tqdm(enumerate(configs), ascii=True, desc='Completed: ', total=len(configs)):
+            config = impute_inactive_values(config).get_array()
+            X_prime.append(config)
+            X_non_hyper.append(config)
+            y_prime.append(self.model.predict_marginalized_over_instances(np.array([X_prime[-1]]))[0].flatten())
+            for idx, param in enumerate(self.scenario.cs.get_hyperparameters()):
+                if not isinstance(param, CategoricalHyperparameter):
+                    X_non_hyper[-1][idx] = param._transform(X_non_hyper[-1][idx])
+        X_non_hyper = np.array(X_non_hyper)
+        X_prime = np.array(X_prime)
+        y_prime = np.array(y_prime)
+        # y_prime = np.array(self.model.predict_marginalized_over_instances(X_prime)[0])
+        self.X = X_prime
+        self.X_fanova = X_non_hyper
+        self.y_fanova = y_prime
+        self.y = y_prime
+        self.logger.info('Size of training X after preprocessing: %s' % str(self.X.shape))
+        self.logger.info('Size of training y after preprocessing: %s' % str(self.y.shape))
+        self.logger.info('Finished Preprocessing')
+        self._preprocessed = True
+
+    def _setup_scenario(self, scenario: Union[None, Scenario], scenario_file: Union[None, str], save_folder: str) -> \
+            None:
+        """
+        Setup for the scenario
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        if scenario is not None:
+            self.scenario = scenario
+        elif scenario_file is not None:
+            self.logger.info('Reading Scenario file and files specified in the scenario')
+            self.scenario = Scenario(scenario=scenario_file)
+            self.scenario.output_dir = save_folder
+            self.scenario.output_dir_for_this_run = save_folder
+            written = self.scenario.out_writer.write_scenario_file(self.scenario)
+        else:
+            raise Exception('Either a scenario has to be given or a file to load it from! Both were set to None!')
+
+    def _load_incumbent(self, traj_file, runhistory_file, incumbent, predict_best=True) -> None:
+        """
+        Handles the loading of the incumbent according to the given parameters.
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        self.incumbent = (None, None)
+        if incumbent is not None:
+            self.incumbent = incumbent
+        elif traj_file is not None:
+            self.logger.info('Reading traj_file: %s' % traj_file)
+            self.incumbent = self._read_traj_file(traj_file)[0]
+            self.logger.debug('Incumbent %s' % str(self.incumbent))
+        elif traj_file is None and runhistory_file is not None:
+            traj_files = os.path.join(os.path.dirname(runhistory_file), 'traj_aclib2.json')
+            traj_files = sorted(glob.glob(traj_files, recursive=True))
+            incumbents = []
+            for traj_ in traj_files:
+                self.logger.info('Reading traj_file: %s' % traj_)
+                incumbents.append(self._read_traj_file(traj_))
+                incumbents[-1].extend(self._model.predict_marginalized_over_instances(
+                    np.array([impute_inactive_values(incumbents[-1][0]).get_array()])))
+                self.logger.debug(incumbents[-1])
+            sort_idx = 2 if predict_best else 1
+            incumbents = sorted(enumerate(incumbents), key=lambda x: x[1][sort_idx])
+            self.best_dir = os.path.dirname(traj_files[incumbents[0][0]])
+            self.incumbent = incumbents[0][1][0]
+            self.logger.info('Incumbent %s' % str(self.incumbent))
+        else:
+            raise Exception('No method specified to load an incumbent. Either give the incumbent directly or specify '
+                            'a file to load it from!')
+
+    def _setup_model(self) -> None:
+        """
+        Sets up all the necessary parameters used for the model.
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        self.logger.info('Converting Data and constructing Model')
+        self.X = None
+        self.y = None
+        self.types = None
+        self.bounds = None
+        self._model = None
+        self.logged_y = False
+        self._convert_data(fit=True)
+        if self.preprocess:
+            self._preprocess(self.runhistory)
+            if self.scenario.run_obj == "runtime":
+                self.y = np.log10(self.y)
+            self.model = 'urfi'
+            self.model.train(self.X, self.y)
+
+    def _load_runhist(self, runhistory, runhistory_file) -> None:
+        """
+        Handels loading of the runhistory/runhistories.
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        self.logger.debug(runhistory_file)
+        self.logger.debug(runhistory)
+        if runhistory is not None:
+            self.runhistory = runhistory
+        elif runhistory_file is not None:
+            self.logger.info('Reading Runhistory')
+            self.runhistory = RunHistory()
+
+            globed_files = glob.glob(runhistory_file)
+            self.logger.info('#RunHistories found: %d' % len(globed_files))
+            if not globed_files:
+                self.logger.error('No runhistory files found!')
+                sys.exit(1)
+            self.runhistory.load_json(globed_files[0], self.scenario.cs)
+            for rh_file in globed_files[1:]:
+                self.runhistory.update_from_json(rh_file, self.scenario.cs)
+        else:
+            raise Exception('Either a runhistory or files to load them from have to be specified! Both were set to '
+                            'None!')
+        self.logger.info('Combined number of Runhistory data points: %d' % len(self.runhistory.data))
+        self.logger.info('Number of Configurations: %d' % (len(self.runhistory.get_all_configs())))
+
+    def _read_traj_file(self, fn):
+        """
+        Simple method to read in a trajectory file in the json format / aclib2 format
+        :param fn:
+            file name
+        :return:
+            tuple of (incumbent [Configuration], incumbent_cost [float])
+        """
+        if not (os.path.exists(fn) and os.path.isfile(fn)):  # File existence check
+            raise FileNotFoundError('File %s not found!' % fn)
+        with open(fn) as fp:
+            # In aclib2, the incumbent is a list of strings, in alljson it's a dictionary.
+            fileformat = 'aclib2' if isinstance(json.loads(fp.readline())["incumbent"], list) else 'alljson'
+
+        if fileformat == "aclib2":
+            self.logger.info("Format is 'aclib2'. This format has issues with recovering configurations properly. We "
+                             "recommend to use the alljson-format.")
+            traj = TrajLogger.read_traj_aclib_format(fn, self.scenario.cs)
+        else:
+            traj = TrajLogger.read_traj_alljson_format(fn, self.scenario.cs)
+
+        incumbent_cost = traj[-1]['cost']
+        incumbent = traj[-1]['incumbent']
+        return [incumbent, incumbent_cost]
+
+    @property
+    def model(self):
+        return self._model
+
+    def _get_types(self, scenario, features):
+        types, bounds = get_types(scenario, features)
+        types = np.array(types, dtype='uint')
+        bounds = np.array(bounds, dtype='object')
+        return types, bounds
+
+    @model.setter
+    def model(self, model_short_name='urfi'):
+        if model_short_name not in ['urfi', 'rfi']:
+            raise ValueError('Specified model %s does not exist or not supported!' % model_short_name)
+        elif model_short_name == 'rfi':
+            self.types, self.bounds = self._get_types(self.scenario.cs, self.scenario.feature_array)
+            self._model = RandomForestWithInstances(self.scenario.cs, self.types, self.bounds, 12345,
+                                                    instance_features=self.scenario.feature_array,
+                                                    logged_y=self.logged_y)
+        elif model_short_name == 'urfi':
+            self.logged_y = True
+            if not self._preprocessed:
+                self.types, self.bounds = self._get_types(self.scenario.cs, self.scenario.feature_array)
+                self._model = UnloggedEPARXrfi(self.scenario.cs, self.types, self.bounds, 12345,
+                                               instance_features=self.scenario.feature_array,
+                                               cutoff=self.cutoff, threshold=self.threshold,
+                                               logged_y=self.logged_y)
+            else:
+                self.types, self.bounds = self._get_types(self.scenario.cs, None)
+                self._model = Unloggedrfwi(self.scenario.cs, self.types, self.bounds, 12345,
+                                           instance_features=None,
+                                           logged_y=self.logged_y)
+        self._model.rf_opts.compute_oob_error = True
+
+    @property
+    def evaluator(self) -> AbstractEvaluator:
+        """
+        Getter of the evaluator property. Returns the set evaluation method.
+        :return: AbstractEvaluator
+        """
+        return self._evaluator
+
+    @evaluator.setter
+    def evaluator(self, evaluation_method: str) -> None:
+        """
+        Setter of the evaluator property. The wanted evaluation method can be specified as string and the rest is
+        handled automatically here
+        :param evaluation_method: Name of the evaluation method to use
+        :return: None
+        """
+        if self._model is None:
+            self._setup_model()
+        self.logger.info('Setting up Evaluation Method')
+        if evaluation_method not in ['ablation', 'fanova', 'forward-selection', 'influence-model',
+                                     'incneighbor', 'lpi']:
+            raise ValueError('Specified evaluation method %s does not exist!' % evaluation_method)
+        if evaluation_method == 'ablation':
+            if self.incumbent is None:
+                raise ValueError('Incumbent is %s!\n \
+                                 Incumbent has to be read from a trajectory file before ablation can be used!'
+                                 % self.incumbent)
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = Ablation(scenario=self.scenario,
+                                 cs=self.scenario.cs,
+                                 model=self._model,
+                                 to_evaluate=self._parameters_to_evaluate,
+                                 incumbent=self.incumbent,
+                                 logy=self.logged_y,
+                                 rng=self.rng,
+                                 verbose=self.verbose)
+        elif evaluation_method == 'influence-model':
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = InfluenceModel(scenario=self.scenario,
+                                       cs=self.scenario.cs,
+                                       model=self._model,
+                                       to_evaluate=self._parameters_to_evaluate,
+                                       margin=self.margin,
+                                       threshold=self.threshold,
+                                       rng=self.rng,
+                                       verbose=self.verbose)
+        elif evaluation_method == 'fanova':
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            mini = None
+            if self.cut_def_fan:
+                mini = True         # TODO what about scenarios where we maximize?
+            evaluator = fANOVA(scenario=self.scenario,
+                               cs=self.scenario.cs,
+                               model=self._model,
+                               to_evaluate=self._parameters_to_evaluate,
+                               runhist=self.runhistory,
+                               rng=self.rng,
+                               minimize=mini,
+                               pairwise=self.pairiwse_fANOVA,
+                               preprocessed_X=self.X_fanova,
+                               preprocessed_y=self.y_fanova,
+                               incumbents=self.incumbent,
+                               verbose=self.verbose)
+        elif evaluation_method in ['incneighbor', 'lpi']:
+            if self.incumbent is None:
+                raise ValueError('Incumbent is %s!\n \
+                                 Incumbent has to be read from a trajectory file before LPI can be used!'
+                                 % self.incumbent)
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = LPI(scenario=self.scenario,
+                            cs=self.scenario.cs,
+                            model=self._model,
+                            to_evaluate=self._parameters_to_evaluate,
+                            incumbent=self.incumbent,
+                            logy=self.logged_y,
+                            rng=self.rng,
+                            quant_var=self.incn_quant_var,
+                            verbose=self.verbose)
+        else:
+            self.logger.info('Using model %s' % str(self.model))
+            evaluator = ForwardSelector(scenario=self.scenario,
+                                        cs=self.scenario.cs,
+                                        model=self._model,
+                                        to_evaluate=self._parameters_to_evaluate,
+                                        rng=self.rng,
+                                        feature_imp=self.forwardsel_feat_imp,
+                                        cv=self.forwardsel_cv,
+                                        verbose=self.verbose)
+        self._evaluator = evaluator
+
+    def _convert_data(self, fit=True) -> None:  # From Marius
+        '''
+            converts data from runhistory into EPM format
+            Parameters
+            ----------
+            scenario: Scenario
+                smac.scenario.scenario.Scenario Object
+            runhistory: RunHistory
+                smac.runhistory.runhistory.RunHistory Object with all necessary data
+            Returns
+            -------
+            np.array
+                X matrix with configuartion x features for all observed samples
+            np.array
+                y matrix with all observations
+            np.array
+                types of X cols -- necessary to train our RF implementation
+        '''
+
+        params = self.scenario.cs.get_hyperparameters()
+        num_params = len(params)
+        self.logger.debug("Counted %d hyperparameters", num_params)
+
+        if self.scenario.run_obj == "runtime":
+            self.cutoff = self.scenario.cutoff
+            self.threshold = self.scenario.cutoff * self.scenario.par_factor
+            self.model = 'urfi'
+            self.logged_y = True
+            # if we log the performance data,
+            # the RFRImputator will already get
+            # log transform data from the runhistory
+            cutoff = np.log10(self.scenario.cutoff)
+            threshold = np.log10(self.scenario.cutoff *
+                                 self.scenario.par_factor)
+            model = RandomForestWithInstances(self.scenario.cs,
+                                              self.types, self.bounds, 12345,
+                                              instance_features=self.scenario.feature_array,
+                                              )
+
+            imputor = RFRImputator(rng=self.rng,
+                                   cutoff=cutoff,
+                                   threshold=threshold,
+                                   model=model,
+                                   change_threshold=0.01,
+                                   max_iter=10)
+            rh2EPM = RunHistory2EPM4LogCost(scenario=self.scenario,
+                                            num_params=num_params,
+                                            success_states=[
+                                                StatusType.SUCCESS, ],
+                                            impute_censored_data=self.impute,
+                                            impute_state=[
+                                                StatusType.TIMEOUT, StatusType.CAPPED],
+                                            imputor=imputor)
+        else:
+            self.model = 'rfi'
+            rh2EPM = RunHistory2EPM4Cost(scenario=self.scenario,
+                                         num_params=num_params,
+                                         success_states=[StatusType.SUCCESS],
+                                         impute_censored_data=self.impute,
+                                         impute_state=None)
+        self.logger.info('Using model %s' % str(self.model))
+        X, Y = rh2EPM.transform(self.runhistory)
+
+        self.X = X
+        self.y = Y
+        self.logger.info('Size of training X: %s' % str(self.X.shape))
+        self.logger.info('Size of training y: %s' % str(self.y.shape))
+        self.logger.info('Data was %s imputed' % ('not' if not self.impute else ''))
+        if not self.impute:
+            self.logger.info('Thus the size of X might be smaller than the datapoints in the RunHistory')
+        if fit:
+            self.logger.info('Fitting Model')
+            self.model.train(X, Y)
+
+    def evaluate_scenario(self, methods, save_folder=None, plot_pyplot=True, plot_bokeh=False) -> Union[
+            Tuple[Dict[str, Dict[str, float]], List[AbstractEvaluator]], Dict[str, Dict[str, float]]]:
+        """
+         the given scenario
+        :param evaluation_method: name of the method to use
+        :param sort_by: int, determines the order (only used if evaluation_method == all)
+            0 => Ablation, fANOVA, Forward Selection
+            1 => Ablation, Forward Selection, fANOVA
+            2 => fANOVA, Forward Selection, Ablation
+            3 => fANOVA, Ablation, Forward Selection
+            4 => Forward Selection, Ablation, fANOVA
+            5 => Forward Selection, fANOVA, Ablation
+        :param plot_pyplot: whether to perform standard matplotlib- plotting
+        :param plot_bokeh: whether to perform advanced bokeh plotting
+        :return: if evaluation all: Tupel of dictionary[evaluation_method] -> importance values, List ov evaluator
+                                    names, ordered according to sort_by
+                 else:
+                      dict[evalution_method] -> importance values
+        """
+        # influence-model currently not supported
+        if not len(methods) >= 1:
+            raise ValueError("Specify at least one method to evaluate the scenario!")
+        fn = os.path.join(save_folder, 'pimp_results.json')
+        load = os.path.exists(fn)
+        dict_ = {}
+        for rnd, method in enumerate(methods):
+            self.logger.info('Running %s' % method)
+            self.evaluator = method
+            dict_[self.evaluator.name.lower()] = self.evaluator.run()
+            self.evaluators.append(self.evaluator)
+            if save_folder and plot_pyplot:
+                self.evaluator.plot_result(os.path.join(save_folder, self.evaluator.name.lower()), show=False)
+            if save_folder and plot_bokeh:
+                self.evaluator.plot_bokeh(os.path.join(save_folder, self.evaluator.name.lower() + "_bokeh"))
+            if load:
+                with open(fn, 'r') as in_file:
+                    doct = json.load(in_file)
+                    for key in doct:
+                        dict_[key] = doct[key]
+            if save_folder:
+                with open(fn, 'w') as out_file:
+                    json.dump(dict_, out_file, sort_keys=True, indent=4, separators=(',', ': '))
+                    load = True
+        return dict_, self.evaluators
\ No newline at end of file
diff --git a/deepcave/evaluators/pending/importance_reworked.py b/deepcave/evaluators/pending/importance_reworked.py
new file mode 100644
index 00000000..8bfcccf7
--- /dev/null
+++ b/deepcave/evaluators/pending/importance_reworked.py
@@ -0,0 +1,432 @@
+from typing import Dict, List, Tuple, Union
+
+import glob
+import json
+import logging
+import os
+import sys
+from collections import OrderedDict
+
+import numpy as np
+from pimp.configspace import (
+    CategoricalHyperparameter,
+    Configuration,
+    impute_inactive_values,
+)
+from pimp.epm.base_epm import RandomForestWithInstances
+from pimp.epm.unlogged_epar_x_rfwi import UnloggedEPARXrfi
+from pimp.epm.unlogged_rfwi import Unloggedrfwi
+from pimp.evaluator.ablation import Ablation
+from pimp.evaluator.fanova import fANOVA
+from pimp.evaluator.forward_selection import AbstractEvaluator, ForwardSelector
+from pimp.evaluator.influence_models import InfluenceModel
+from pimp.evaluator.local_parameter_importance import LPI
+from pimp.utils import RunHistory, RunHistory2EPM4Cost, RunHistory2EPM4LogCost, Scenario
+from smac.epm.rfr_imputator import RFRImputator
+from smac.epm.util_funcs import get_types
+from smac.tae import StatusType
+from smac.utils.io.traj_logging import TrajLogger
+from tqdm import tqdm
+
+__author__ = "Andre Biedenkapp"
+__copyright__ = "Copyright 2016, ML4AAD"
+__license__ = "3-clause BSD"
+__maintainer__ = "Andre Biedenkapp"
+__email__ = "biedenka@cs.uni-freiburg.de"
+
+
+class Importance(object):
+    def __init__(self,
+                 objective: str,
+                 runhistory: RunHistory,
+                 configspace: ConfigSpace):
+
+        self.logger = logging.getLogger("Importance")
+
+        self.objective = objective
+        self.runhistory
+        self.configspace = configspace
+
+        self._setup_model()
+        self._load_incumbent()
+ 
+
+    def _preprocess(self, runhistory):
+        """
+        Method to marginalize over instances such that fANOVA can determine the parameter importance without
+        having to deal with instance features.
+        :param runhistory: RunHistory that knows all configurations that were run. For all these configurations
+                           we have to marginalize away the instance features with which fANOVA will make it's
+                           predictions
+        """
+        self.logger.info('PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING')
+        self.logger.info('Marginalizing away all instances!')
+        configs = runhistory.get_all_configs()
+        X_non_hyper, X_prime, y_prime = [], [], []
+        for c_id, config in tqdm(enumerate(configs), ascii=True, desc='Completed: ', total=len(configs)):
+            config = impute_inactive_values(config).get_array()
+            X_prime.append(config)
+            X_non_hyper.append(config)
+            y_prime.append(self.model.predict_marginalized_over_instances(np.array([X_prime[-1]]))[0].flatten())
+            for idx, param in enumerate(self.scenario.cs.get_hyperparameters()):
+                if not isinstance(param, CategoricalHyperparameter):
+                    X_non_hyper[-1][idx] = param._transform(X_non_hyper[-1][idx])
+        X_non_hyper = np.array(X_non_hyper)
+        X_prime = np.array(X_prime)
+        y_prime = np.array(y_prime)
+        # y_prime = np.array(self.model.predict_marginalized_over_instances(X_prime)[0])
+        self.X = X_prime
+        self.X_fanova = X_non_hyper
+        self.y_fanova = y_prime
+        self.y = y_prime
+        self.logger.info('Size of training X after preprocessing: %s' % str(self.X.shape))
+        self.logger.info('Size of training y after preprocessing: %s' % str(self.y.shape))
+        self.logger.info('Finished Preprocessing')
+        self._preprocessed = True
+
+    def _setup_scenario(self, scenario: Union[None, Scenario], scenario_file: Union[None, str], save_folder: str) -> \
+            None:
+        """
+        Setup for the scenario
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        if scenario is not None:
+            self.scenario = scenario
+        elif scenario_file is not None:
+            self.logger.info('Reading Scenario file and files specified in the scenario')
+            self.scenario = Scenario(scenario=scenario_file)
+            self.scenario.output_dir = save_folder
+            self.scenario.output_dir_for_this_run = save_folder
+            written = self.scenario.out_writer.write_scenario_file(self.scenario)
+        else:
+            raise Exception('Either a scenario has to be given or a file to load it from! Both were set to None!')
+
+    def _load_incumbent(self, traj_file, runhistory_file, incumbent, predict_best=True) -> None:
+        """
+        Handles the loading of the incumbent according to the given parameters.
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        self.incumbent = (None, None)
+        if incumbent is not None:
+            self.incumbent = incumbent
+        elif traj_file is not None:
+            self.logger.info('Reading traj_file: %s' % traj_file)
+            self.incumbent = self._read_traj_file(traj_file)[0]
+            self.logger.debug('Incumbent %s' % str(self.incumbent))
+        elif traj_file is None and runhistory_file is not None:
+            traj_files = os.path.join(os.path.dirname(runhistory_file), 'traj_aclib2.json')
+            traj_files = sorted(glob.glob(traj_files, recursive=True))
+            incumbents = []
+            for traj_ in traj_files:
+                self.logger.info('Reading traj_file: %s' % traj_)
+                incumbents.append(self._read_traj_file(traj_))
+                incumbents[-1].extend(self._model.predict_marginalized_over_instances(
+                    np.array([impute_inactive_values(incumbents[-1][0]).get_array()])))
+                self.logger.debug(incumbents[-1])
+            sort_idx = 2 if predict_best else 1
+            incumbents = sorted(enumerate(incumbents), key=lambda x: x[1][sort_idx])
+            self.best_dir = os.path.dirname(traj_files[incumbents[0][0]])
+            self.incumbent = incumbents[0][1][0]
+            self.logger.info('Incumbent %s' % str(self.incumbent))
+        else:
+            raise Exception('No method specified to load an incumbent. Either give the incumbent directly or specify '
+                            'a file to load it from!')
+
+    def _setup_model(self) -> None:
+        """
+        Sets up all the necessary parameters used for the model.
+        Helper method to have the init method less cluttered.
+        For parameter specifications, see __init__
+        """
+        self.logger.info('Converting Data and constructing Model')
+        self.X = None
+        self.y = None
+        self.types = None
+        self.bounds = None
+        self._model = None
+        self.logged_y = False
+
+        self._convert_data(fit=True)
+
+
+        if self.preprocess:
+            self._preprocess(self.runhistory)
+            if self.scenario.run_obj == "runtime":
+                self.y = np.log10(self.y)
+            self.model = 'urfi'
+            self.model.train(self.X, self.y)
+
+
+    def _read_traj_file(self, fn):
+        """
+        Simple method to read in a trajectory file in the json format / aclib2 format
+        :param fn:
+            file name
+        :return:
+            tuple of (incumbent [Configuration], incumbent_cost [float])
+        """
+        if not (os.path.exists(fn) and os.path.isfile(fn)):  # File existence check
+            raise FileNotFoundError('File %s not found!' % fn)
+        with open(fn) as fp:
+            # In aclib2, the incumbent is a list of strings, in alljson it's a dictionary.
+            fileformat = 'aclib2' if isinstance(json.loads(fp.readline())["incumbent"], list) else 'alljson'
+
+        if fileformat == "aclib2":
+            self.logger.info("Format is 'aclib2'. This format has issues with recovering configurations properly. We "
+                             "recommend to use the alljson-format.")
+            traj = TrajLogger.read_traj_aclib_format(fn, self.scenario.cs)
+        else:
+            traj = TrajLogger.read_traj_alljson_format(fn, self.scenario.cs)
+
+        incumbent_cost = traj[-1]['cost']
+        incumbent = traj[-1]['incumbent']
+        return [incumbent, incumbent_cost]
+
+    @property
+    def model(self):
+        return self._model
+
+    def _get_types(self, scenario, features):
+        types, bounds = get_types(scenario, features)
+        types = np.array(types, dtype='uint')
+        bounds = np.array(bounds, dtype='object')
+        return types, bounds
+
+    @model.setter
+    def model(self, model_short_name='urfi'):
+        if model_short_name not in ['urfi', 'rfi']:
+            raise ValueError('Specified model %s does not exist or not supported!' % model_short_name)
+        elif model_short_name == 'rfi':
+            self.types, self.bounds = self._get_types(self.scenario.cs, self.scenario.feature_array)
+            self._model = RandomForestWithInstances(self.scenario.cs, self.types, self.bounds, 12345,
+                                                    instance_features=self.scenario.feature_array,
+                                                    logged_y=self.logged_y)
+        elif model_short_name == 'urfi':
+            self.logged_y = True
+            if not self._preprocessed:
+                self.types, self.bounds = self._get_types(self.scenario.cs, self.scenario.feature_array)
+                self._model = UnloggedEPARXrfi(self.scenario.cs, self.types, self.bounds, 12345,
+                                               instance_features=self.scenario.feature_array,
+                                               cutoff=self.cutoff, threshold=self.threshold,
+                                               logged_y=self.logged_y)
+            else:
+                self.types, self.bounds = self._get_types(self.scenario.cs, None)
+                self._model = Unloggedrfwi(self.scenario.cs, self.types, self.bounds, 12345,
+                                           instance_features=None,
+                                           logged_y=self.logged_y)
+        self._model.rf_opts.compute_oob_error = True
+
+    @property
+    def evaluator(self) -> AbstractEvaluator:
+        """
+        Getter of the evaluator property. Returns the set evaluation method.
+        :return: AbstractEvaluator
+        """
+        return self._evaluator
+
+    @evaluator.setter
+    def evaluator(self, evaluation_method: str) -> None:
+        """
+        Setter of the evaluator property. The wanted evaluation method can be specified as string and the rest is
+        handled automatically here
+        :param evaluation_method: Name of the evaluation method to use
+        :return: None
+        """
+        if self._model is None:
+            self._setup_model()
+        self.logger.info('Setting up Evaluation Method')
+        if evaluation_method not in ['ablation', 'fanova', 'forward-selection', 'influence-model',
+                                     'incneighbor', 'lpi']:
+            raise ValueError('Specified evaluation method %s does not exist!' % evaluation_method)
+        if evaluation_method == 'ablation':
+            if self.incumbent is None:
+                raise ValueError('Incumbent is %s!\n \
+                                 Incumbent has to be read from a trajectory file before ablation can be used!'
+                                 % self.incumbent)
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = Ablation(scenario=self.scenario,
+                                 cs=self.scenario.cs,
+                                 model=self._model,
+                                 to_evaluate=self._parameters_to_evaluate,
+                                 incumbent=self.incumbent,
+                                 logy=self.logged_y,
+                                 rng=self.rng,
+                                 verbose=self.verbose)
+        elif evaluation_method == 'influence-model':
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = InfluenceModel(scenario=self.scenario,
+                                       cs=self.scenario.cs,
+                                       model=self._model,
+                                       to_evaluate=self._parameters_to_evaluate,
+                                       margin=self.margin,
+                                       threshold=self.threshold,
+                                       rng=self.rng,
+                                       verbose=self.verbose)
+        elif evaluation_method == 'fanova':
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            mini = None
+            if self.cut_def_fan:
+                mini = True         # TODO what about scenarios where we maximize?
+            evaluator = fANOVA(scenario=self.scenario,
+                               cs=self.scenario.cs,
+                               model=self._model,
+                               to_evaluate=self._parameters_to_evaluate,
+                               runhist=self.runhistory,
+                               rng=self.rng,
+                               minimize=mini,
+                               pairwise=self.pairiwse_fANOVA,
+                               preprocessed_X=self.X_fanova,
+                               preprocessed_y=self.y_fanova,
+                               incumbents=self.incumbent,
+                               verbose=self.verbose)
+        elif evaluation_method in ['incneighbor', 'lpi']:
+            if self.incumbent is None:
+                raise ValueError('Incumbent is %s!\n \
+                                 Incumbent has to be read from a trajectory file before LPI can be used!'
+                                 % self.incumbent)
+            self.logger.info('Using model %s' % str(self.model))
+            self.logger.info('X shape %s' % str(self.model.X.shape))
+            evaluator = LPI(scenario=self.scenario,
+                            cs=self.scenario.cs,
+                            model=self._model,
+                            to_evaluate=self._parameters_to_evaluate,
+                            incumbent=self.incumbent,
+                            logy=self.logged_y,
+                            rng=self.rng,
+                            quant_var=self.incn_quant_var,
+                            verbose=self.verbose)
+        else:
+            self.logger.info('Using model %s' % str(self.model))
+            evaluator = ForwardSelector(scenario=self.scenario,
+                                        cs=self.scenario.cs,
+                                        model=self._model,
+                                        to_evaluate=self._parameters_to_evaluate,
+                                        rng=self.rng,
+                                        feature_imp=self.forwardsel_feat_imp,
+                                        cv=self.forwardsel_cv,
+                                        verbose=self.verbose)
+        self._evaluator = evaluator
+
+    def _convert_data(self, fit=True) -> None:  # From Marius
+        '''
+            converts data from runhistory into EPM format
+            Parameters
+            ----------
+            scenario: Scenario
+                smac.scenario.scenario.Scenario Object
+            runhistory: RunHistory
+                smac.runhistory.runhistory.RunHistory Object with all necessary data
+            Returns
+            -------
+            np.array
+                X matrix with configuartion x features for all observed samples
+            np.array
+                y matrix with all observations
+            np.array
+                types of X cols -- necessary to train our RF implementation
+        '''
+
+        params = self.scenario.cs.get_hyperparameters()
+        num_params = len(params)
+        self.logger.debug("Counted %d hyperparameters", num_params)
+
+        if self.scenario.run_obj == "runtime":
+            self.cutoff = self.scenario.cutoff
+            self.threshold = self.scenario.cutoff * self.scenario.par_factor
+            self.model = 'urfi'
+            self.logged_y = True
+            # if we log the performance data,
+            # the RFRImputator will already get
+            # log transform data from the runhistory
+            cutoff = np.log10(self.scenario.cutoff)
+            threshold = np.log10(self.scenario.cutoff *
+                                 self.scenario.par_factor)
+            model = RandomForestWithInstances(self.scenario.cs,
+                                              self.types, self.bounds, 12345,
+                                              instance_features=self.scenario.feature_array,
+                                              )
+
+            imputor = RFRImputator(rng=self.rng,
+                                   cutoff=cutoff,
+                                   threshold=threshold,
+                                   model=model,
+                                   change_threshold=0.01,
+                                   max_iter=10)
+            rh2EPM = RunHistory2EPM4LogCost(scenario=self.scenario,
+                                            num_params=num_params,
+                                            success_states=[
+                                                StatusType.SUCCESS, ],
+                                            impute_censored_data=self.impute,
+                                            impute_state=[
+                                                StatusType.TIMEOUT, StatusType.CAPPED],
+                                            imputor=imputor)
+        else:
+            self.model = 'rfi'
+            rh2EPM = RunHistory2EPM4Cost(scenario=self.scenario,
+                                         num_params=num_params,
+                                         success_states=[StatusType.SUCCESS],
+                                         impute_censored_data=self.impute,
+                                         impute_state=None)
+        self.logger.info('Using model %s' % str(self.model))
+        X, Y = rh2EPM.transform(self.runhistory)
+
+        self.X = X
+        self.y = Y
+        self.logger.info('Size of training X: %s' % str(self.X.shape))
+        self.logger.info('Size of training y: %s' % str(self.y.shape))
+        self.logger.info('Data was %s imputed' % ('not' if not self.impute else ''))
+        if not self.impute:
+            self.logger.info('Thus the size of X might be smaller than the datapoints in the RunHistory')
+        if fit:
+            self.logger.info('Fitting Model')
+            self.model.train(X, Y)
+
+    def evaluate_scenario(self, methods, save_folder=None, plot_pyplot=True, plot_bokeh=False) -> Union[
+            Tuple[Dict[str, Dict[str, float]], List[AbstractEvaluator]], Dict[str, Dict[str, float]]]:
+        """
+         the given scenario
+        :param evaluation_method: name of the method to use
+        :param sort_by: int, determines the order (only used if evaluation_method == all)
+            0 => Ablation, fANOVA, Forward Selection
+            1 => Ablation, Forward Selection, fANOVA
+            2 => fANOVA, Forward Selection, Ablation
+            3 => fANOVA, Ablation, Forward Selection
+            4 => Forward Selection, Ablation, fANOVA
+            5 => Forward Selection, fANOVA, Ablation
+        :param plot_pyplot: whether to perform standard matplotlib- plotting
+        :param plot_bokeh: whether to perform advanced bokeh plotting
+        :return: if evaluation all: Tupel of dictionary[evaluation_method] -> importance values, List ov evaluator
+                                    names, ordered according to sort_by
+                 else:
+                      dict[evalution_method] -> importance values
+        """
+        # influence-model currently not supported
+        if not len(methods) >= 1:
+            raise ValueError("Specify at least one method to evaluate the scenario!")
+        fn = os.path.join(save_folder, 'pimp_results.json')
+        load = os.path.exists(fn)
+        dict_ = {}
+        for rnd, method in enumerate(methods):
+            self.logger.info('Running %s' % method)
+            self.evaluator = method
+            dict_[self.evaluator.name.lower()] = self.evaluator.run()
+            self.evaluators.append(self.evaluator)
+            if save_folder and plot_pyplot:
+                self.evaluator.plot_result(os.path.join(save_folder, self.evaluator.name.lower()), show=False)
+            if save_folder and plot_bokeh:
+                self.evaluator.plot_bokeh(os.path.join(save_folder, self.evaluator.name.lower() + "_bokeh"))
+            if load:
+                with open(fn, 'r') as in_file:
+                    doct = json.load(in_file)
+                    for key in doct:
+                        dict_[key] = doct[key]
+            if save_folder:
+                with open(fn, 'w') as out_file:
+                    json.dump(dict_, out_file, sort_keys=True, indent=4, separators=(',', ': '))
+                    load = True
+        return dict_, self.evaluators
\ No newline at end of file
diff --git a/deepcave/evaluators/pending/lpi.py b/deepcave/evaluators/pending/lpi.py
new file mode 100644
index 00000000..a3e7caeb
--- /dev/null
+++ b/deepcave/evaluators/pending/lpi.py
@@ -0,0 +1,125 @@
+from deepcave.evaluators.epm.rf_with_instances import RandomForestWithInstances
+
+
+class LPI:
+    def __init__(self, run: Run):
+        # fidelity=None,
+        # states=None,
+        # hyperparameter_ids=None):
+
+        self.incumbent = run.get_incumbent(fidelity, hyperparameter_ids)
+        #self.transform_config = lambda config, mapping=mapping: run.transform_config(config, mapping)
+
+        X, y, mapping, id_mapping, types, bounds, configspace = run.transform_configs(
+            self,
+            fidelities,
+            states,
+            hyperparameters
+        )
+
+        self.cs = configspace
+
+        # We need a new configspace
+
+        # Initialize model
+        self.model = RandomForestWithInstances(
+            configspace,
+            types,
+            bounds,
+            do_bootstrapping=True
+        )
+        self.model.rf_opts.compute_oob_error = True
+        self.model.train(X, y)
+
+        pass
+
+    def _get_one_exchange_neighborhood_by_parameter(self):
+        """
+        Slight modification of ConfigSpace's get_one_exchange neighborhood. This orders the parameter values and samples
+        more neighbors in one go. Further we need to rigorously check each and every neighbor if it is forbidden or not.
+        """
+
+        neighborhood_dict = {}
+        hp_names = list(self.incumbent.keys())
+
+        for hp_id, hp_name in enumerate(hp_names):
+            self.logger.info('Sampling neighborhood of %s' % hp_name)
+
+            array = self.incumbent.get_array()
+            # array = self.transform_config(self.incumbent)
+
+            if not np.isfinite(array[hp_id]):
+                self.logger.info('>'.join(['-'*50, ' Not active!']))
+                continue
+
+            neighbourhood = []
+            checked_neighbors = []
+            checked_neighbors_non_unit_cube = []
+            hp = self.incumbent.configuration_space.get_hyperparameter(hp_name)
+            num_neighbors = hp.get_num_neighbors(self.incumbent.get(hp_name))
+
+            self.logger.debug('\t' + str(num_neighbors))
+            if num_neighbors == 0:
+                self.logger.debug('\tNo neighbors!')
+                continue
+            elif np.isinf(num_neighbors):  # Continous Parameters
+                if hp.log:
+                    base = np.e
+                    log_lower = np.log(hp.lower) / np.log(base)
+                    log_upper = np.log(hp.upper) / np.log(base)
+                    neighbors = np.logspace(log_lower, log_upper, self.continous_neighbors,
+                                            endpoint=True, base=base)
+                else:
+                    neighbors = np.linspace(
+                        hp.lower, hp.upper, self.continous_neighbors)
+                neighbors = list(
+                    map(lambda x: hp._inverse_transform(x), neighbors))
+            else:
+                neighbors = hp.get_neighbors(array[hp_id], self.rng)
+
+            for neighbor in neighbors:
+                if neighbor in checked_neighbors:
+                    continue
+
+                new_array = array.copy()
+                new_array = change_hp_value(
+                    self.incumbent.configuration_space,
+                    new_array,
+                    param,
+                    neighbor,
+                    index
+                )
+
+                try:
+                    new_configuration = Configuration(
+                        self.incumbent.configuration_space, vector=new_array)
+                    neighbourhood.append(new_configuration)
+                    new_configuration.is_valid_configuration()
+                    check_forbidden(self.cs.forbidden_clauses, new_array)
+                    checked_neighbors.append(neighbor)
+                    checked_neighbors_non_unit_cube.append(
+                        new_configuration[param])
+                except (ForbiddenValueError, ValueError) as e:
+                    pass
+
+            self.logger.info('>'.join(
+                ['-'*50, ' Found {:>3d} valid neighbors'.format(len(checked_neighbors))]))
+            self.sampled_neighbors += len(checked_neighbors) + 1
+            sort_idx = list(map(lambda x: x[0], sorted(
+                enumerate(checked_neighbors), key=lambda y: y[1])))
+            if isinstance(self.cs.get_hyperparameter(param), CategoricalHyperparameter):
+                checked_neighbors_non_unit_cube = list(
+                    np.array(checked_neighbors_non_unit_cube)[sort_idx])
+            else:
+                checked_neighbors_non_unit_cube = np.array(
+                    checked_neighbors_non_unit_cube)[sort_idx]
+            neighborhood_dict[param] = [np.array(checked_neighbors)[
+                sort_idx], checked_neighbors_non_unit_cube]
+
+        return neighborhood_dict
+
+    def train(self, X, y):
+        pass
+
+    def run(self):
+        neighborhood_dict = self._get_one_exchange_neighborhood_by_parameter()
diff --git a/deepcave/evaluators/pending/lpi_old.py b/deepcave/evaluators/pending/lpi_old.py
new file mode 100644
index 00000000..6c455681
--- /dev/null
+++ b/deepcave/evaluators/pending/lpi_old.py
@@ -0,0 +1,234 @@
+import os
+from collections import OrderedDict
+from copy import deepcopy
+
+import numpy as np
+from bokeh.models import Panel, Row, Tabs
+from tqdm import tqdm
+
+
+class LPI(AbstractEvaluator):
+
+    """
+    Implementation of Ablation via surrogates
+    """
+
+    def __init__(self, scenario, cs, model, to_evaluate: int, incumbent=None, continous_neighbors=500, quant_var=True, **kwargs):
+        super().__init__(scenario, cs, model, to_evaluate, **kwargs)
+
+        self.incumbent = incumbent
+        self.incumbent_dict = self.incumbent.get_dictionary()
+        self.continous_neighbors = continous_neighbors
+        self.neighborhood_dict = None
+        self.performance_dict = {}
+        self.sampled_neighbors = 0
+        self.variance_dict = {}
+        self.quantify_importance_via_variance = quant_var
+        self.evaluated_parameter_importance_uncertainty = OrderedDict()
+
+    def _get_one_exchange_neighborhood_by_parameter(self):
+        """
+        Slight modification of ConfigSpace's get_one_exchange neighborhood. This orders the parameter values and samples
+        more neighbors in one go. Further we need to rigorously check each and every neighbor if it is forbidden or not.
+        """
+        neighborhood_dict = {}
+        params = list(self.incumbent.keys())
+        self.logger.debug('params: ' + str(params))
+        for index, param in enumerate(params):
+            self.logger.info('Sampling neighborhood of %s' % param)
+            array = self.incumbent.get_array()
+
+            if not np.isfinite(array[index]):
+                self.logger.info('>'.join(['-'*50, ' Not active!']))
+                continue
+
+            neighbourhood = []
+            checked_neighbors = []
+            checked_neighbors_non_unit_cube = []
+            hp = self.incumbent.configuration_space.get_hyperparameter(param)
+            num_neighbors = hp.get_num_neighbors(self.incumbent.get(param))
+            self.logger.debug('\t' + str(num_neighbors))
+            if num_neighbors == 0:
+                self.logger.debug('\tNo neighbors!')
+                continue
+            elif np.isinf(num_neighbors):  # Continous Parameters
+                if hp.log:
+                    base = np.e
+                    log_lower = np.log(hp.lower) / np.log(base)
+                    log_upper = np.log(hp.upper) / np.log(base)
+                    neighbors = np.logspace(log_lower, log_upper, self.continous_neighbors,
+                                            endpoint=True, base=base)
+                else:
+                    neighbors = np.linspace(hp.lower, hp.upper, self.continous_neighbors)
+                neighbors = list(map(lambda x: hp._inverse_transform(x), neighbors))
+            else:
+                neighbors = hp.get_neighbors(array[index], self.rng)
+            for neighbor in neighbors:
+                if neighbor in checked_neighbors:
+                    continue
+                new_array = array.copy()
+                new_array = change_hp_value(self.incumbent.configuration_space, new_array, param, neighbor,
+                                            index)
+                try:
+                    new_configuration = Configuration(self.incumbent.configuration_space, vector=new_array)
+                    neighbourhood.append(new_configuration)
+                    new_configuration.is_valid_configuration()
+                    check_forbidden(self.cs.forbidden_clauses, new_array)
+                    checked_neighbors.append(neighbor)
+                    checked_neighbors_non_unit_cube.append(new_configuration[param])
+                except (ForbiddenValueError, ValueError) as e:
+                    pass
+
+            self.logger.info('>'.join(['-'*50, ' Found {:>3d} valid neighbors'.format(len(checked_neighbors))]))
+            self.sampled_neighbors += len(checked_neighbors) + 1
+            sort_idx = list(map(lambda x: x[0], sorted(enumerate(checked_neighbors), key=lambda y: y[1])))
+            if isinstance(self.cs.get_hyperparameter(param), CategoricalHyperparameter):
+                checked_neighbors_non_unit_cube = list(np.array(checked_neighbors_non_unit_cube)[sort_idx])
+            else:
+                checked_neighbors_non_unit_cube = np.array(checked_neighbors_non_unit_cube)[sort_idx]
+            neighborhood_dict[param] = [np.array(checked_neighbors)[sort_idx], checked_neighbors_non_unit_cube]
+
+        return neighborhood_dict
+
+    def _predict_over_instance_set(self, config):
+        """
+        Small wrapper to predict marginalized over instances
+        Parameter
+        ---------
+        config:Configuration
+            The self.incumbent of wich the performance across the whole instance set is to be estimated
+        Returns
+        -------
+        mean
+            the mean performance over the instance set
+        var
+            the variance over the instance set. If logged values are used, the variance might not be able to be used
+        """
+        mean, var = self.model.predict_marginalized_over_instances(np.array([config.get_array()]))
+        return mean.squeeze(), var.squeeze()
+
+    def run(self) -> OrderedDict:
+        """
+        Main function.
+        Returns
+        -------
+        evaluated_parameter_importance:OrderedDict
+            Parameter -> importance. The order is important as smaller indices indicate higher importance
+        """
+        neighborhood_dict = self._get_one_exchange_neighborhood_by_parameter()  # sampled on a unit-hypercube!
+        self.neighborhood_dict = neighborhood_dict
+        incumbent_array = self.incumbent.get_array()
+        def_perf, def_var = self._predict_over_instance_set(impute_inactive_values(self.cs.get_default_configuration()))
+        inc_perf, inc_var = self._predict_over_instance_set(impute_inactive_values(self.incumbent))
+        delta = def_perf - inc_perf
+        evaluated_parameter_importance = {}
+
+        # These are used for plotting and hold the predictions for each neighbor of each parameter
+        # That means performance_dict holds the mean, variance_dict the variance of the forest
+        performance_dict = {}
+        variance_dict = {}
+        # This are used for importance and hold the corresponding importance/variance over neighbors
+        # Only import if NOT quantifying importance via performance-variance across neighbours
+        overall_imp = {}
+        # Nested list of values per tree in random forest
+        pred_per_tree = {}
+
+        # Iterate over parameters
+        pbar = tqdm(range(self.sampled_neighbors), ascii=True, disable=not self.verbose)
+        for index, param in enumerate(self.incumbent.keys()):
+            if param not in neighborhood_dict:
+                pbar.set_description('{: >.70s}'.format('Parameter %s is inactive' % param))
+                continue
+
+            pbar.set_description('Predicting performances for neighbors of {: >.30s}'.format(param))
+            performance_dict[param] = []
+            variance_dict[param] = []
+            pred_per_tree[param] = []
+            added_inc = False
+            inc_at = 0
+            # Iterate over neighbors
+            for unit_neighbor, neighbor in zip(neighborhood_dict[param][0], neighborhood_dict[param][1]):
+                if not added_inc:
+                    # Detect incumbent
+                    if unit_neighbor > incumbent_array[index]:
+                        performance_dict[param].append(inc_perf)
+                        variance_dict[param].append(inc_var)
+                        pbar.update(1)
+                        added_inc = True
+                    else:
+                        inc_at += 1
+                # self.logger.debug('%s -> %s' % (self.incumbent[param], neighbor))
+                # Create the neighbor-Configuration object
+                new_array = incumbent_array.copy()
+                new_array = change_hp_value(self.incumbent.configuration_space, new_array,
+                                            param, unit_neighbor, index)
+                new_configuration = impute_inactive_values(Configuration(self.incumbent.configuration_space,
+                                                                         vector=new_array))
+                # Predict performance
+                x = np.array(new_configuration.get_array())
+                pred_per_tree[param].append([np.mean(tree_pred) for tree_pred in self.model.rf.all_leaf_values(x)])
+                # self.logger.debug("Pred per tree: %s", str(pred_per_tree[param][-1]))
+                performance_dict[param].append(np.mean(pred_per_tree[param][-1]))
+                variance_dict[param].append(np.var(pred_per_tree[param][-1]))
+
+                pbar.update(1)
+            if len(neighborhood_dict[param][0]) > 0:
+                neighborhood_dict[param][0] = np.insert(neighborhood_dict[param][0], inc_at, incumbent_array[index])
+                neighborhood_dict[param][1] = np.insert(neighborhood_dict[param][1], inc_at, self.incumbent[param])
+            else:
+                neighborhood_dict[param][0] = np.array(incumbent_array[index])
+                neighborhood_dict[param][1] = [self.incumbent[param]]
+            if not added_inc:
+                mean, var = self._predict_over_instance_set(impute_inactive_values(self.incumbent))
+                performance_dict[param].append(mean)
+                variance_dict[param].append(var)
+                pbar.update(1)
+            # After all neighbors are estimated, look at all performances except the incumbent
+            tmp_perf = performance_dict[param][:inc_at] + performance_dict[param][inc_at + 1:]
+            if delta == 0:
+                delta = 1  # To avoid division by zero
+            imp_over_mea = (np.mean(tmp_perf) - performance_dict[param][inc_at]) / delta
+            imp_over_med = (np.median(tmp_perf) - performance_dict[param][inc_at]) / delta
+            try:
+                imp_over_max = (np.max(tmp_perf) - performance_dict[param][inc_at]) / delta
+            except ValueError:
+                imp_over_max = np.nan  # Hacky fix as this is never used anyway
+            overall_imp[param] = np.array([imp_over_mea, imp_over_med, imp_over_max])
+
+        # Creating actual importance value (by normalizing over sum of vars)
+        num_trees = len(list(pred_per_tree.values())[0][0])
+        params = list(performance_dict.keys())
+        overall_var_per_tree = {
+                param : [np.var([neighbor[tree_idx] for neighbor in pred_per_tree[param]])
+                            for tree_idx in range(num_trees)]
+                                for param in params}
+        # Sum up variances per tree across parameters
+        sum_var_per_tree = [sum([overall_var_per_tree[param][tree_idx] for param in params])
+                                     for tree_idx in range(num_trees)]
+        # Normalize
+        overall_var_per_tree = {p : [t / sum_var_per_tree[idx] for idx, t in enumerate(trees)] for p, trees in
+                                                    overall_var_per_tree.items()}
+        self.logger.debug("overall_var_per_tree %s (%d trees)",
+                str(overall_var_per_tree), len(list(pred_per_tree.values())[0][0]))
+        self.logger.debug("sum_var_per_tree %s (%d trees)",
+                str(sum_var_per_tree), len(list(pred_per_tree.values())[0][0]))
+        for param in performance_dict.keys():
+            if self.quantify_importance_via_variance:
+                evaluated_parameter_importance[param] = np.mean(overall_var_per_tree[param])
+            else:
+                evaluated_parameter_importance[param] = overall_imp[param][0]
+
+        only_show = sorted(list(evaluated_parameter_importance.keys()),
+                           key=lambda p: evaluated_parameter_importance[p])[:min(self.to_evaluate,
+                                                                                 len(evaluated_parameter_importance.keys()))]
+
+        self.neighborhood_dict = neighborhood_dict
+        self.performance_dict = performance_dict
+        self.variance_dict = variance_dict
+        self.evaluated_parameter_importance = OrderedDict([(p, evaluated_parameter_importance[p]) for p in only_show])
+        if self.quantify_importance_via_variance:
+            self.evaluated_parameter_importance_uncertainty = OrderedDict([(p, np.std(overall_var_per_tree[p])) for p in only_show])
+        all_res = {'imp': self.evaluated_parameter_importance,
+                   'order': list(self.evaluated_parameter_importance.keys())}
+                   
+        return all_res
\ No newline at end of file
diff --git a/deepcave/layouts/__init__.py b/deepcave/layouts/__init__.py
index e69de29b..566a5554 100644
--- a/deepcave/layouts/__init__.py
+++ b/deepcave/layouts/__init__.py
@@ -0,0 +1,19 @@
+from abc import ABC, abstractmethod
+from typing import Union
+
+from dash.development.base_component import Component
+
+from deepcave.utils.logs import get_logger
+
+
+class Layout(ABC):
+    def __init__(self):
+        self.register_callbacks()
+        self.logger = get_logger(self.__class__.__name__)
+
+    def register_callbacks(self):
+        pass
+
+    @abstractmethod
+    def __call__(self) -> Union[list[Component], Component]:
+        pass
diff --git a/deepcave/layouts/general.py b/deepcave/layouts/general.py
index 08908eed..88d526ad 100644
--- a/deepcave/layouts/general.py
+++ b/deepcave/layouts/general.py
@@ -1,151 +1,166 @@
-import os
+from typing import Type
+
+from pathlib import Path
 
-from dash import dcc
-from dash import html
 import dash_bootstrap_components as dbc
-from dash.dependencies import Input, Output, State, ALL
+from dash import dcc, html
+from dash.dependencies import ALL, Input, Output, State
+from dash.development.base_component import Component
 from dash.exceptions import PreventUpdate
 
-from deepcave import app, c, rc
-from deepcave.config import CONFIG
-from deepcave.runs.handler import handler
-from deepcave.layouts.layout import Layout
-from deepcave.utils.hash import string_to_hash
+from deepcave import app, c, rc, run_handler
+from deepcave.layouts import Layout
+from deepcave.runs import NotMergeableError, NotValidRunError
+from deepcave.runs.run import Run
 
 
 class GeneralLayout(Layout):
-    def register_callbacks(self):
+    def register_callbacks(self) -> None:
+        self._callback_working_directory_changed()
+        self._callback_run_selection_changed()
+        self._callback_group_selection_changed()
+        self._callback_set_groups()
+        self._callback_clear_cache()
+
+    def _callback_working_directory_changed(self) -> None:
         outputs = [
-            Output('general-working-directory-input', 'value'),
-            Output('general-converter-label', 'children'),
-            Output('general-runs-checklist', 'options'),
-            Output('general-runs-checklist', 'value'),
+            # Working directory input
+            Output("general-working-directory-input", "value"),
+            # Converter text
+            Output("general-converter-label", "children"),
+            Output("general-runs-checklist", "options"),
+            Output("general-runs-checklist", "value"),
         ]
 
         inputs = [
-            Input('on-page-load', 'href'),
-            Input('general-working-directory-input', 'value'),
+            Input("on-page-load", "href"),
+            Input("general-working-directory-input", "value"),
         ]
 
         # Register updates from inputs
         @app.callback(outputs, inputs)
-        def general_update(_, working_dir):
+        def callback(_, working_dir: str):
             # `working_dir` is only none on page load
             if working_dir is None:
-                working_dir = handler.get_working_dir()
-                run_ids = handler.get_run_ids()
-                converter = handler.get_converter()
-
-                return \
-                    working_dir, \
-                    self.get_converter_text(converter), \
-                    self.get_run_options(), \
-                    list(run_ids.keys())
-
-            # Check if working dir exists
-            if working_dir is None or not os.path.isdir(working_dir):
-                PreventUpdate()
-
-            run_names = []
-            handler.set_working_dir(working_dir)
-            handler.set_run_names(run_names)  # Reset run ids
-            handler.set_groups({})  # Reset groups
-
-            # Find converter name
-            converter = handler.get_converter()
-            if converter is None:
-                PreventUpdate()
-
-            return \
-                working_dir, \
-                self.get_converter_text(converter), \
-                self.get_run_options(), \
-                run_names
-
-        input = Input('general-runs-checklist', 'value')
-        output = Output('general-run-names', 'value')
+                handler_working_dir = run_handler.get_working_dir()
+                run_names = run_handler.get_run_names()
+                converter = run_handler.available_run_classes
+
+                return (
+                    str(handler_working_dir),
+                    self.get_converter_text(converter),
+                    self.get_run_options(),
+                    run_names,
+                )
+
+            empty_run_names = []
+            run_handler.update_working_directory(Path(working_dir))
+
+            return (
+                working_dir,
+                self.get_converter_text(run_handler.available_run_classes),
+                self.get_run_options(),
+                empty_run_names,
+            )
+
+    def _callback_run_selection_changed(self) -> None:
+        output = Output("general-run-names", "data")
+        input = Input("general-runs-checklist", "value")
 
         # Save the run ids internally
         @app.callback(output, input)
-        def general_register_runs(run_names):
-            old_run_names = handler.get_run_names()
+        def callback(run_ids: list[str]):
+            old_run_names = run_handler.get_run_names()
+            self.logger.debug(f"Old runs: {old_run_names}, Runs: {run_ids}")
 
             # Reset groups here.
             # Alternatively: Remove all runs which are not selected anymore.
-            if run_names != old_run_names:
-                handler.set_groups({})
+            if run_ids != old_run_names:
+                run_handler.update_groups({})
 
                 # Also really important:
                 # Reset last inputs
                 c.set("last_inputs", value={})
 
-            handler.set_run_names(run_names)
-            return run_names
+            try:
+                run_handler.update_runs(run_ids)
+            except NotValidRunError:
+                return old_run_names
+
+            return run_ids
 
+    def _callback_group_selection_changed(self) -> None:
         outputs = [
-            Output('general-group-container', 'children'),
-            Output('general-add-group', 'n_clicks'),
+            Output("general-group-container", "children"),
+            Output("general-add-group", "n_clicks"),
         ]
         inputs = [
-            Input('general-add-group', 'n_clicks'),
-            Input('general-run-names', 'value'),
-            State('general-group-container', 'children')
+            Input("general-add-group", "n_clicks"),
+            Input("general-run-names", "data"),
+            Input("general-group-trigger", "data"),
+            State("general-group-container", "children"),
         ]
 
         # Let's take care of the groups here
         @app.callback(outputs, inputs)
-        def general_display_groups(n_clicks, run_names, children):
-            def get_layout(index, options, input_value="", dropdown_value=[]):
-                return html.Div([
-                    dbc.Input(
-                        id={'type': 'group-name', 'index': index},
-                        placeholder="Name",
-                        type="text",
-                        value=input_value,
-                        style={"margin-bottom": "-1px"}),
-                    dcc.Dropdown(
-                        id={'type': 'group-dropdown', 'index': index},
-                        options=[{'label': i, 'value': i} for i in options],
-                        value=dropdown_value,
-                        multi=True,
-                    )
-                ], className="mb-2")
-
-            groups = handler.get_groups()
+        def callback(n_clicks: int, run_names, update, children):
+            def get_layout(index, options, input_value="", dropdown_value=None):
+                if dropdown_value is None:
+                    dropdown_value = []
+                return html.Div(
+                    [
+                        dbc.Input(
+                            id={"type": "group-name", "index": index},
+                            placeholder="Name",
+                            type="text",
+                            value=input_value,
+                            style={"margin-bottom": "-1px"},
+                        ),
+                        dcc.Dropdown(
+                            id={"type": "group-dropdown", "index": index},
+                            options=[
+                                {"label": name, "value": name} for name in options
+                            ],
+                            value=dropdown_value,
+                            multi=True,
+                        ),
+                    ],
+                    className="mb-2",
+                )
+
+            groups = run_handler.get_groups()
             index = 0
 
             # Load from cache if page is loaded
             children = []
-            for group_name, selected_run_names in groups.items():
+            for group_name, grouped_run in groups.items():
                 if group_name is None:
                     continue
 
                 children.append(
-                    get_layout(index, run_names, group_name,
-                               selected_run_names)
+                    get_layout(index, run_names, group_name, grouped_run.run_names)
                 )
 
                 index += 1
 
             if n_clicks is not None and len(run_names) > 0:
-                children.append(
-                    get_layout(index, run_names)
-                )
+                children.append(get_layout(index, run_names))
 
             return children, None
 
-        outputs = Output('general-group-output', 'data')
+    def _callback_set_groups(self) -> None:
+        outputs = Output("general-group-trigger", "data")
         inputs = [
-            Input({'type': 'group-name', 'index': ALL}, 'value'),
-            Input({'type': 'group-dropdown', 'index': ALL}, 'value')
+            Input({"type": "group-name", "index": ALL}, "value"),
+            Input({"type": "group-dropdown", "index": ALL}, "value"),
         ]
 
         @app.callback(outputs, inputs)
-        def general_set_groups(group_names, all_run_names):
+        def callback(group_names, all_run_names):
             # Abort on page load
             if self._refresh_groups:
                 self._refresh_groups = False
-                return
+                raise PreventUpdate()
 
             groups = {}
             for group_name, run_names in zip(group_names, all_run_names):
@@ -154,77 +169,73 @@ def general_set_groups(group_names, all_run_names):
 
                 if run_names is None or len(run_names) == 0:
                     continue
-                    # run_names = []
 
                 groups[group_name] = run_names
 
-            # Now save it
-            handler.set_groups(groups)
+            try:
+                # Now save it
+                run_handler.update_groups(groups)
+            except NotMergeableError:
+                return True
 
-            return
+            self.logger.debug(f"Groups: {groups}")
 
-        output = Output('general-clear-cache-button', 'n_clicks')
-        input = Input('general-clear-cache-button', 'n_clicks')
+            raise PreventUpdate()
+
+    def _callback_clear_cache(self) -> None:
+        output = Output("general-clear-cache-button", "n_clicks")
+        input = Input("general-clear-cache-button", "n_clicks")
 
         @app.callback(output, input)
-        def general_clear_cache(n_clicks):
+        def callback(n_clicks):
             if n_clicks is not None:
-                rc.clear_all()
+                rc.clear_all_caches()
 
             return None
 
-    @ staticmethod
-    def get_run_options():
-        return [{"label": run_name, "value": run_name} for run_name in handler.get_available_run_names()]
-
-    @ staticmethod
-    def get_converter_text(converter):
-        converter_text = ""
-        if converter is not None:
-            converter_text = [
-                html.Div([
-                    html.Span("Found compatible runs from "),
-                    html.I(converter.name()),
-                    html.Span(".")
-                ], className="mt-2")
-            ]
-
-        return converter_text
-
-    def __call__(self):
+    @staticmethod
+    def get_run_options() -> list[dict[str, str]]:
+        runs = [
+            {"label": run_name, "value": run_name}
+            for run_name in run_handler.get_available_run_names()
+        ]
+        return runs
+
+    @staticmethod
+    def get_converter_text(converters: list[Type[Run]]) -> html.Div:
+        converter_text = []
+        for converter in converters:
+            converter_text += [converter.prefix]
+
+        return html.Div(
+            html.Span(f"Available run converters: {', '.join(converter_text)}"),
+            className="mt-2",
+        )
+
+    def __call__(self) -> list[Component]:
         self._refresh_groups = True
 
         return [
-            html.H1('General'),
-
+            html.H1("General"),
             dbc.Label("Working Directory"),
-            #html.Div("Working Directory"),
-            #dbc.FormText("Absolute path to your runs."),
-            dbc.Input(id="general-working-directory-input",
-                      placeholder="",
-                      type="text"),
-
+            # html.Div("Working Directory"),
+            # dbc.FormText("Absolute path to your runs."),
+            dbc.Input(
+                id="general-working-directory-input", placeholder="", type="text"
+            ),
             dbc.FormText(id="general-converter-label"),
-
             html.Hr(),
-
-            html.H2('Runs'),
-            dbc.Input(id="general-run-names", style={"display": "none"}),
+            html.H2("Runs"),
+            dcc.Store(id="general-run-names"),
             dbc.Checklist(id="general-runs-checklist"),
-
             html.Hr(),
-
-            html.H2('Groups'),
+            html.H2("Groups"),
             html.Div(id="general-group-container", children=[]),
             dbc.Button("Add Group", id="general-add-group"),
-            dcc.Store(id="general-group-output"),
-
+            dcc.Store(id="general-group-trigger"),
             html.Hr(),
-
-            html.H2('Caches'),
-            dbc.Button("Clear Plugin Caches",
-                       id="general-clear-cache-button", color="primary"),
+            html.H2("Caches"),
+            dbc.Button(
+                "Clear Plugin Caches", id="general-clear-cache-button", color="primary"
+            ),
         ]
-
-
-layout = GeneralLayout()
diff --git a/deepcave/layouts/header.py b/deepcave/layouts/header.py
index 6fec58c5..d32020cc 100644
--- a/deepcave/layouts/header.py
+++ b/deepcave/layouts/header.py
@@ -1,11 +1,9 @@
-from dash import html
 import dash_bootstrap_components as dbc
-from dash import dcc
+from dash import dcc, html
 from dash.dependencies import Input, Output
 
-from deepcave.layouts.layout import Layout
-from deepcave import app
-from deepcave import c
+from deepcave import app, c
+from deepcave.layouts import Layout
 
 
 class HeaderLayout(Layout):
@@ -13,13 +11,13 @@ def register_callbacks(self):
         super().register_callbacks()
 
         outputs = [
-            Output('matplotlib-mode-toggle', 'color'),
-            Output('matplotlib-mode-badge', 'children'),
-            Output('matplotlib-mode-refresh', 'href'),
+            Output("matplotlib-mode-toggle", "color"),
+            Output("matplotlib-mode-badge", "children"),
+            Output("matplotlib-mode-refresh", "href"),
         ]
         inputs = [
-            Input('matplotlib-mode-toggle', 'n_clicks'),
-            Input('matplotlib-mode-refresh', 'pathname'),
+            Input("matplotlib-mode-toggle", "n_clicks"),
+            Input("matplotlib-mode-refresh", "pathname"),
         ]
 
         @app.callback(outputs, inputs)
@@ -39,22 +37,29 @@ def update_matplotlib_mode(n_clicks, pathname):
             else:
                 return "secondary", "off", update
 
-    def __call__(self):
-        return html.Header(className='navbar navbar-dark sticky-top bg-dark flex-md-nowrap p-0 shadow', children=[
-            html.A(
-                'DeepCave', className='navbar-brand me-0 px-3', href='#'),
-            html.Button(
-                className='navbar-toggler position-absolute d-md-none collapsed'),
-
-            dcc.Location(id="matplotlib-mode-refresh", refresh=True),
-            dbc.Button(
-                ["Matplotlib", dbc.Badge(
-                    "off", color="light", text_color="black", className="ms-2", id="matplotlib-mode-badge")],
-                color="secondary",
-                className="me-2",
-                id="matplotlib-mode-toggle"
-            ),
-        ])
-
-
-layout = HeaderLayout()
+    def __call__(self) -> html.Header:
+        return html.Header(
+            className="navbar navbar-dark sticky-top bg-dark flex-md-nowrap p-0 shadow",
+            children=[
+                html.A("DeepCave", className="navbar-brand me-0 px-3", href="#"),
+                html.Button(
+                    className="navbar-toggler position-absolute d-md-none collapsed"
+                ),
+                dcc.Location(id="matplotlib-mode-refresh", refresh=True),
+                dbc.Button(
+                    [
+                        "Matplotlib",
+                        dbc.Badge(
+                            "off",
+                            color="light",
+                            text_color="black",
+                            className="ms-2",
+                            id="matplotlib-mode-badge",
+                        ),
+                    ],
+                    color="secondary",
+                    className="me-2",
+                    id="matplotlib-mode-toggle",
+                ),
+            ],
+        )
diff --git a/deepcave/layouts/layout.py b/deepcave/layouts/layout.py
deleted file mode 100644
index 2da8ab24..00000000
--- a/deepcave/layouts/layout.py
+++ /dev/null
@@ -1,13 +0,0 @@
-from abc import ABC, abstractmethod
-
-
-class Layout:
-    def __init__(self):
-        self.register_callbacks()
-
-    def register_callbacks(self):
-        pass
-
-    @abstractmethod
-    def __call__(self):
-        pass
diff --git a/deepcave/layouts/main.py b/deepcave/layouts/main.py
index e58a6192..8ab80c15 100644
--- a/deepcave/layouts/main.py
+++ b/deepcave/layouts/main.py
@@ -1,65 +1,82 @@
-from dash import dcc
-from dash import html
-import dash_bootstrap_components as dbc
-from dash.dependencies import Input, Output, State
+from dash import dcc, html
+from dash.dependencies import Input, Output
+from dash.development.base_component import Component
 
-from deepcave import app, queue
-from deepcave.runs.handler import handler
-from deepcave.layouts.layout import Layout
-from deepcave.layouts.header import layout as header_layout
-from deepcave.layouts.general import layout as general_layout
-from deepcave.layouts.not_found import layout as not_found_layout
-from deepcave.layouts.sidebar import layout as sidebar_layout
-from deepcave.plugins import plugin_layouts
+from deepcave import app, queue, run_handler
+from deepcave.layouts import Layout
+from deepcave.layouts.general import GeneralLayout
+from deepcave.layouts.header import HeaderLayout
+from deepcave.layouts.not_found import NotFoundLayout
+from deepcave.layouts.sidebar import SidebarLayout
+from deepcave.plugins import Plugin
+from deepcave.utils.dash import alert
 
 
 class MainLayout(Layout):
-    def __init__(self):
+    def __init__(self, categorized_plugins: dict[str, list[Plugin]]):
         super().__init__()
+        self.plugins = {}
+        self.sidebar_layout = SidebarLayout(categorized_plugins)
+        self.header_layout = HeaderLayout()
+        self.general_layout = GeneralLayout()
+        self.not_found_layout = NotFoundLayout  # Needs to be class here to add url
+        for plugins in categorized_plugins.values():
+            for plugin in plugins:
+                self.plugins[plugin.id] = plugin
 
     def register_callbacks(self):
-        output = Output('content', 'children')
-        input = Input('on-page-load', 'pathname')
+        output = Output("content", "children")
+        input = Input("on-page-load", "pathname")
 
         @app.callback(output, input)
-        def display_page(pathname):
+        def display_page(pathname: str):
             paths = pathname.split("/")[1:]
 
             if paths[0] == "":
-                return general_layout()
+                return self.general_layout()
             else:
-                def alert(text): return dbc.Alert(
-                    text, id="alert", is_open=True, dismissable=False, fade=True, color="danger")
-
                 if not queue.ready():
                     return alert("At least one worker has to be enabled.")
-                if len(handler.get_run_names()) == 0:
+                if len(run_handler.get_run_names()) == 0:
                     return alert("Please select a run first.")
                 else:
                     if paths[0] == "plugins":
-                        for name, layout in plugin_layouts.items():
-                            if name == paths[1]:
-                                return layout()
-
-            return not_found_layout
-
-    def __call__(self):
-        return \
-            html.Div(children=[
-                header_layout(),
-                html.Div(id='main-container', className='container-fluid', children=[
-                    html.Div(className='row', children=[
-                        sidebar_layout(),
-                        html.Main(className='col-md-9 ms-sm-auto col-lg-10 px-md-4', children=[
-                            html.Div(className='', children=[
-                                dcc.Location(id='on-page-load', refresh=False),
-                                html.Div(id='content'),
-                                html.Div(id='general_message')
-                            ])
-                        ])
-                    ]),
-                ]),
-            ])
+                        plugin = self.plugins.get(paths[1], None)
+                        if plugin is not None:
+                            return plugin()
 
+            return self.not_found_layout(pathname)()
 
-layout = MainLayout()
+    def __call__(self) -> Component:
+        return html.Div(
+            children=[
+                self.header_layout(),
+                html.Div(
+                    id="main-container",
+                    className="container-fluid",
+                    children=[
+                        html.Div(
+                            className="row",
+                            children=[
+                                self.sidebar_layout(),
+                                html.Main(
+                                    className="col-md-9 ms-sm-auto col-lg-10 px-md-4",
+                                    children=[
+                                        html.Div(
+                                            className="",
+                                            children=[
+                                                dcc.Location(
+                                                    id="on-page-load", refresh=False
+                                                ),
+                                                html.Div(id="content"),
+                                                html.Div(id="general_message"),
+                                            ],
+                                        )
+                                    ],
+                                ),
+                            ],
+                        ),
+                    ],
+                ),
+            ]
+        )
diff --git a/deepcave/layouts/not_found.py b/deepcave/layouts/not_found.py
index 8cb6aaf7..cc80bc41 100644
--- a/deepcave/layouts/not_found.py
+++ b/deepcave/layouts/not_found.py
@@ -1,12 +1,16 @@
 from dash import html
+from dash.development.base_component import Component
 
-#from deepcave.server.plugins.plugin_manager import PluginManager
-from deepcave.layouts.layout import Layout
+from deepcave.layouts import Layout
 
 
 class NotFoundLayout(Layout):
-    def __call__(self):
-        return html.H2('This page does not exists.')
+    def __init__(self, url):
+        super().__init__()
+        self.url = url
 
-
-layout = NotFoundLayout()
+    def __call__(self) -> list[Component]:
+        return [
+            html.H2("This page does not exists."),
+            html.Div(f"Tried to reach {self.url}"),
+        ]
diff --git a/deepcave/layouts/sidebar.py b/deepcave/layouts/sidebar.py
index 866d3ff9..f429953b 100644
--- a/deepcave/layouts/sidebar.py
+++ b/deepcave/layouts/sidebar.py
@@ -1,42 +1,39 @@
-from dash import html
-from dash.dependencies import Input, Output, State
-from dash.exceptions import PreventUpdate
-from dash import dcc
+from typing import Union
+
+from dash import dcc, html
+from dash.dependencies import Input, Output
+from dash.development.base_component import Component
 
 from deepcave import app, queue
-from deepcave.layouts.layout import Layout
-from deepcave.plugins import plugin_names, plugin_categories
+from deepcave.layouts import Layout
+from deepcave.plugins import Plugin
 
 
 class SidebarLayout(Layout):
-
-    def __init__(self):
+    def __init__(self, categorized_plugins: dict[str, list[Plugin]]):
         super().__init__()
+        self.plugins = categorized_plugins
 
-        nav_points = {}
-        for id, name in plugin_names.items():
-
-            category = plugin_categories[id]
-
-            if category not in nav_points:
-                nav_points[category] = []
-
-            nav_points[category].append((id, name))
+        nav_points = {category: [] for category in categorized_plugins}
+        for category, plugins in categorized_plugins.items():
+            for plugin in plugins:
+                nav_points[category].append((plugin.id, plugin.name, plugin.icon))
 
         self.nav_points = nav_points
 
     def register_callbacks(self):
         # Update queue information panel
         @app.callback(
-            Output("queue-info", 'children'),
-            Input("queue-info-interval", 'n_intervals'))
+            Output("queue-info", "children"),
+            Input("queue-info-interval", "n_intervals"),
+        )
         def update_queue_info(_):
             try:
                 jobs = {}
 
                 for job in queue.get_running_jobs():
-                    display_name = job.meta['display_name']
-                    run_name = job.meta['run_name']
+                    display_name = job.meta["display_name"]
+                    run_name = job.meta["run_name"]
 
                     if display_name not in jobs:
                         jobs[display_name] = []
@@ -44,8 +41,8 @@ def update_queue_info(_):
                     jobs[display_name].append((run_name, "[R]"))
 
                 for job in queue.get_pending_jobs():
-                    display_name = job.meta['display_name']
-                    run_name = job.meta['run_name']
+                    display_name = job.meta["display_name"]
+                    run_name = job.meta["run_name"]
 
                     if display_name not in jobs:
                         jobs[display_name] = []
@@ -54,73 +51,92 @@ def update_queue_info(_):
 
                 items = []
                 for display_name, run_names in jobs.items():
-                    items += [html.Li(
-                        className='nav-item',
-                        children=[
-                            html.A(f"{display_name}",
-                                   className='nav-link')
-                        ]
-                    )]
+                    items += [
+                        html.Li(
+                            className="nav-item",
+                            children=[html.A(f"{display_name}", className="nav-link")],
+                        )
+                    ]
 
                     for run_name, status in run_names:
                         items += [
                             html.Li(
-                                className='nav-item',
+                                className="nav-item",
                                 children=[
-                                    html.A(f"{status} {run_name}",
-                                           className='nav-link disabled',
-                                           style={'padding-top': 0, 'padding-bottom': 0})
-                                ]
+                                    html.A(
+                                        f"{status} {run_name}",
+                                        className="nav-link disabled",
+                                        style={"padding-top": 0, "padding-bottom": 0},
+                                    )
+                                ],
                             )
                         ]
 
                 if len(jobs) > 0:
                     return [
                         html.Hr(),
-                        html.H6(className='sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted', children=[
-                            html.Span("Queue Information")
-                        ]),
-                        html.Ul(className='nav flex-column', children=items),
+                        html.H6(
+                            className="sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted",
+                            children=[html.Span("Queue Information")],
+                        ),
+                        html.Ul(className="nav flex-column", children=items),
                     ]
 
                 return []
             except:
                 return
 
-    def __call__(self):
+    def __call__(self) -> Union[list[Component], Component]:
 
         layouts = []
         for category, points in self.nav_points.items():
             layouts += [
-                html.H6(className='sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted', children=[
-                    html.Span(category)
-                ])
+                html.H6(
+                    className="sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted",
+                    children=[html.Span(category)],
+                )
             ]
 
             point_layouts = []
-            for (id, name) in points:
-                point_layouts += [html.Li(
-                    className='nav-item',
-                    children=[html.A(name, className='nav-link active', href=f'/plugins/{id}')])
+            for (id, name, icon) in points:
+                point_layouts += [
+                    html.Li(
+                        className="nav-item",
+                        children=[
+                            html.A(
+                                [html.I(className=icon), name],
+                                className="nav-link",
+                                href=f"/plugins/{id}",
+                            )
+                        ],
+                    )
                 ]
 
-            layouts += [html.Ul(className='nav flex-column',
-                                children=point_layouts)]
-
-        return \
-            html.Nav(className='col-md-3 col-lg-2 d-md-block sidebar collapse', id='sidebarMenu', children=[
-                html.Div(className='position-sticky pt-3', children=[
-                    html.Ul(className='nav flex-column', children=[
-                        html.A("General", className='nav-link active', href='/'),
-                    ]),
-
-                    *layouts
-
-                ]),
-
+            layouts += [html.Ul(className="nav flex-column", children=point_layouts)]
+
+        icon = {"data-feather": "file-text"}
+
+        return html.Nav(
+            className="col-md-3 col-lg-2 d-md-block sidebar collapse",
+            id="sidebarMenu",
+            children=[
+                html.Div(
+                    className="position-sticky pt-3",
+                    children=[
+                        html.Ul(
+                            className="nav flex-column",
+                            children=[
+                                html.A(
+                                    "General",
+                                    className="nav-link active",
+                                    href="/",
+                                ),
+                            ],
+                        ),
+                        *layouts,
+                    ],
+                ),
                 dcc.Interval(id="queue-info-interval", interval=1000),
-                html.Div(id="queue-info")
-            ])
-
-
-layout = SidebarLayout()
+                html.Div(id="queue-info"),
+            ],
+        )
diff --git a/deepcave/plugins/__init__.py b/deepcave/plugins/__init__.py
index 99771243..d711f37f 100644
--- a/deepcave/plugins/__init__.py
+++ b/deepcave/plugins/__init__.py
@@ -1,72 +1,683 @@
-import os
+from abc import ABC, abstractmethod
+from typing import Any, Callable, Iterable, Optional, Union, List
 
-from deepcave.plugins.plugin import Plugin
-from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.plugins.static_plugin import StaticPlugin
-from deepcave.utils.importing import auto_import_iter
+import copy
 
+import dash_bootstrap_components as dbc
+from dash import dcc, html
+from dash.dash import no_update
+from dash.dependencies import Input, Output
+from dash.development.base_component import Component
+from dash.exceptions import PreventUpdate
+from pytest import raises
 
-def read_plugins():
-    plugins = {}
+from deepcave import app, c, run_handler
+from deepcave.layouts import Layout
+from deepcave.runs import AbstractRun
+from deepcave.runs.grouped_run import GroupedRun, NotMergeableError
+from deepcave.runs.run import Run
+from deepcave.utils.data_structures import update_dict
+from deepcave.utils.layout import get_select_options
+from deepcave.utils.logs import get_logger
+from deepcave.utils.util import add_prefix_to_dict
 
-    paths = [os.path.join(os.path.abspath(os.curdir),
-                          'deepcave', 'plugins', '*')]
+logger = get_logger(__name__)
 
-    for _, obj in auto_import_iter("deepcave.plugins", paths):
 
-        if issubclass(obj, Plugin):
-            # Plugin itself is a subclass, filter it out
-            if obj.__module__ == DynamicPlugin.__module__ or \
-               obj.__module__ == StaticPlugin.__module__ or \
-               obj.__module__ == Plugin.__module__:
-                continue
+class Plugin(Layout, ABC):
+    id: str
+    name: str
+    category: Optional[str] = None
+    description: Optional[str] = None
+    position: int = 99999
+    icon: str = "far fa-file"
 
-            pos = obj.position()
-            if pos not in plugins:
-                plugins[pos] = []
+    button_caption: str = "Process"
 
-            plugins[pos].append(obj)
+    """
+    activate_run_selection:
+    Shows a dropdown to select a run in the inputs layout. This feature is useful if only one run could be viewed at
+    a time. Moreover, it prevents the plugin to calculate results across all runs.
 
-    plugins_ordered = []
-    for pos in sorted(plugins):
-        plugins_ordered += plugins[pos]
+    The run can be selected by inputs["run_name"]["value"].
+    bool: True if run selection should be shown.
+    """
+    activate_run_selection: bool = False
 
-    return plugins_ordered
+    def __init__(self):
+        self.inputs = []
+        self.outputs = []
 
+        # Processing right now?
+        self.blocked = False
 
-plugins = read_plugins()
+        # Alert texts
+        self.alert_text = ""
+        self.alert_color = "success"
+        self.alert_update_required = False
 
+        self.runs: dict[
+            str, AbstractRun
+        ] = {}  # Set in __call__: run_name -> AbstractRun
 
-def get_plugin_layouts():
-    layouts = {}
+        super().__init__()
 
-    for plugin in plugins:
-        layouts[plugin.id()] = plugin()
+    @staticmethod
+    def check_run_compatibility(run: AbstractRun) -> bool:
+        """
+        Checks if a run is compatible with this plugin. If a plugin is not compatible,
+        you can not select the run.
 
-    return layouts
+        Notes
+        -----
+        This function is only called if `activate_run_selection` is True.
 
+        Parameters
+        ----------
+        run : AbstractRun
+            One of the selected runs/groups.
 
-def get_plugin_names():
-    names = {}
+        Returns
+        -------
+        bool
+            Returns True if the run is compatible.
 
-    for plugin in plugins:
-        names[plugin.id()] = plugin.name()
+        """
 
-    return names
+        return True
 
+    def check_runs_compatibility(self, runs: List[AbstractRun]) -> None:
+        """
+        This function is needed if all selected runs need something in common
+        (e.g. budget or objective). Since this function is called before the layout is created,
+        it can be also used to set common values for the plugin.
 
-def get_plugin_categories():
-    categories = {}
+        Parameters
+        ----------
+        runs : List[AbstractRun]
+            Selected runs.
 
-    for plugin in plugins:
-        categories[plugin.id()] = plugin.category()
+        Raises
+        ------
+        NotMergeableError
+            If runs are not compatible, an error is thrown.
+        """
 
-    return categories
+        return True
 
+    def register_input(
+        self, id: str, attributes: Union[str, Iterable[str]] = ("value",), filter=False
+    ) -> str:
+        if isinstance(attributes, str):
+            attributes = [attributes]
 
-plugin_layouts = get_plugin_layouts()
-plugin_names = get_plugin_names()
-plugin_categories = get_plugin_categories()
+        for attribute in attributes:
+            key = (id, attribute, filter)
+            if key not in self.inputs:
+                self.inputs.append(key)
 
+        # We have to rearrange the inputs because `State`
+        # must follow all `Input`. Since all filters are `Input`, we have to
+        # shift them to the front.
+        self.inputs.sort(key=lambda x: x[2], reverse=True)
 
-__all__ = [plugin_layouts, plugin_names, plugin_categories]
+        return self.get_internal_input_id(id)
+
+    def register_output(self, id: str, attribute: str = "value", mpl=False) -> str:
+        assert isinstance(attribute, str)
+
+        if mpl:
+            id += "-mpl"
+
+        key = (id, attribute, mpl)
+        if key not in self.outputs:
+            self.outputs.append(key)
+
+        return self.get_internal_output_id(id)
+
+    def get_internal_id(self, id: str) -> str:
+        return f"{self.id}-{id}"
+
+    def get_internal_input_id(self, id: str) -> str:
+        return f"{self.id}-{id}-input"
+
+    def get_internal_output_id(self, id: str) -> str:
+        return f"{self.id}-{id}-output"
+
+    def register_callbacks(self):
+        # We have to call the output layout one time to register
+        # the values
+        # Problem: Inputs/Outputs can't be changed afterwards anymore.
+
+        if self.activate_run_selection:
+            self.__class__.get_run_input_layout(self.register_input)
+
+        self.__class__.get_input_layout(self.register_input)
+        self.__class__.get_filter_layout(
+            lambda a, b: self.register_input(a, b, filter=True)
+        )
+        self.__class__.get_output_layout(self.register_output)
+        self.__class__.get_mpl_output_layout(
+            lambda a, b: self.register_output(a, b, mpl=True)
+        )
+
+        # Handles the initial and the cashed input values
+        outputs = []
+        # [Input(self.get_internal_id("update-button"), 'n_clicks')]
+        inputs = []
+
+        # Define also inputs if they are declared as interactive
+        for id, attribute, _ in self.inputs:
+            inputs.append(Input(self.get_internal_input_id(id), attribute))
+
+        for id, attribute, _ in self.inputs:
+            outputs.append(Output(self.get_internal_input_id(id), attribute))
+
+        if len(outputs) > 0:
+
+            @app.callback(outputs, inputs)
+            def plugin_input_update(*inputs_list):
+                init = True
+                # Simple check if page was loaded for the first time
+                for input in inputs_list:
+                    if input is not None:
+                        init = False
+                        break
+
+                # Reload our inputs
+                if init:
+                    inputs = c.get("last_inputs", self.id)
+
+                    if inputs is None:
+                        inputs = self.load_inputs()
+
+                        # Also update the run selection
+                        if self.activate_run_selection:
+                            new_inputs = self.__class__.load_run_inputs(
+                                self.runs,
+                                self.groups,
+                                self.__class__.check_run_compatibility,
+                            )
+                            update_dict(inputs, new_inputs)
+
+                        # Set not used inputs
+                        for (id, attribute, _) in self.inputs:
+                            if id not in inputs:
+                                inputs[id] = {}
+
+                            if attribute not in inputs[id]:
+                                inputs[id][attribute] = None
+                else:
+                    # Map the list `inputs` to a dict.
+                    inputs = self._list_to_dict(inputs_list)
+
+                    if len(self.previous_inputs) == 0:
+                        self.previous_inputs = inputs.copy()
+
+                    # Only work on copies.
+                    # We don't want the inputs dict to be changed by the user.
+                    _previous_inputs = self.previous_inputs.copy()
+                    _inputs = inputs.copy()
+
+                    selected_run: Optional[AbstractRun] = None
+                    if self.activate_run_selection:
+                        if "run_name" in _previous_inputs:
+                            _previous_run_name = _previous_inputs["run_name"]["value"]
+                        else:
+                            _previous_run_name = None
+                        _run_name = inputs["run_name"]["value"]
+
+                        # Reset everything if run name changed.
+                        if (
+                            _previous_run_name is not None
+                            and _previous_run_name != _run_name
+                        ):
+                            # We can't use load_inputs here only
+                            # because `run_name` would be removed.
+                            # Also: We want to keep the current run name.
+                            update_dict(_inputs, self.load_inputs())
+
+                            # TODO: Reset only inputs which are not available in another ru.
+                            # E.g. if options from budget in run_2 and run_3 are the same
+                            # take the budget from run_2 if changed to run_3. Otherwise, reset budgets.
+
+                        selected_run = self.all_runs[inputs["run_name"]["value"]]
+
+                    # How to update only parameters which have a dependency?
+                    user_dependencies_inputs = self.load_dependency_inputs(
+                        _previous_inputs, _inputs, selected_run
+                    )
+
+                    # Update dict
+                    # dict.update() remove keys, so it's done manually
+                    update_dict(inputs, user_dependencies_inputs)  # inplace
+
+                # From dict to list
+                inputs_list = self._dict_to_list(inputs, input=True)
+                self.previous_inputs = inputs
+
+                return inputs_list
+
+        # Update internal alert state to divs
+        @app.callback(
+            Output(self.get_internal_id("alert"), "children"),
+            Output(self.get_internal_id("alert"), "color"),
+            Output(self.get_internal_id("alert"), "is_open"),
+            Input(self.get_internal_id("alert-interval"), "n_intervals"),
+        )
+        def update_alert(_):
+            if self.alert_update_required:
+                self.alert_update_required = False
+                return self.alert_text, self.alert_color, True
+            else:
+                raise PreventUpdate()
+
+    def update_alert(self, text: str, color: str = "success"):
+        self.alert_text = text
+        self.alert_color = color
+        self.alert_update_required = True
+
+    def _inputs_changed(self, inputs, last_inputs):
+        # Check if last_inputs are the same as the given inputs.
+        inputs_changed = False
+        filters_changed = False
+
+        # If only filters changed, then we don't need to
+        # calculate the results again.
+        if last_inputs is not None:
+            for (id, attribute, filter) in self.inputs:
+
+                if self.activate_run_selection:
+                    if id == "run_name":
+                        continue
+
+                if inputs[id][attribute] != last_inputs[id][attribute]:
+                    if not filter:
+                        inputs_changed = True
+                    else:
+                        filters_changed = True
+
+        return inputs_changed, filters_changed
+
+    def _process_raw_outputs(self, inputs, raw_outputs):
+        logger.debug("Process raw outputs.")
+
+        # Use raw outputs to update our layout
+        mpl_active = c.get("matplotlib-mode")
+        if mpl_active:
+            outputs = self.__class__.load_mpl_outputs(
+                inputs, raw_outputs, self.all_runs
+            )
+        else:
+            outputs = self.__class__.load_outputs(inputs, raw_outputs, self.all_runs)
+
+        if outputs == PreventUpdate:
+            raise PreventUpdate()
+
+        # Map outputs here because it may be that the outputs are
+        # differently sorted than the values were registered.
+        if isinstance(outputs, dict):
+            outputs = self._dict_to_list(outputs, input=False)
+        else:
+            if not isinstance(outputs, list):
+                outputs = [outputs]
+
+        # We have to add no_updates here for the mode we don't want
+        count_outputs = 0
+        count_mpl_outputs = 0
+        for (_, _, mpl_mode) in self.outputs:
+            if mpl_mode:
+                count_mpl_outputs += 1
+            else:
+                count_outputs += 1
+
+        if mpl_active:
+            outputs = [no_update for _ in range(count_outputs)] + outputs
+        else:
+            outputs = outputs + [no_update for _ in range(count_mpl_outputs)]
+
+        return outputs
+
+    def _list_to_dict(
+        self, values: Iterable[str], input=True
+    ) -> dict[str, dict[str, str]]:
+        """
+        Maps the given values to a dict, regarding the sorting from
+        either self.inputs or self.outputs.
+
+        Returns:
+            dict
+        """
+
+        if input:
+            order = self.inputs
+        else:
+            order = self.outputs
+
+        mapping = {}
+        for value, (id, attribute, *_) in zip(values, order):
+            if id not in mapping:
+                mapping[id] = {}
+
+            mapping[id][attribute] = value
+
+        return mapping
+
+    def _dict_to_list(
+        self, d: dict[str, dict[str, str]], input=False
+    ) -> list[Optional[str]]:
+        """
+        Maps the given dict to a list, regarding the sorting from either
+        self.inputs or self.outputs.
+
+        Returns:
+            list.
+        """
+
+        if input:
+            order = self.inputs
+        else:
+            order = self.outputs
+
+        result = []
+        for (id, attribute, instance) in order:
+
+            if not input:
+                # Instance is mlp_mode in case of outputs
+                # Simply ignore other outputs.
+                if instance != c.get("matplotlib-mode"):
+                    continue
+            try:
+                value = d[id][attribute]
+                result += [value]
+            except:
+                result += [None]
+
+        return result
+
+    def _dict_as_key(self, d: dict[str, Any], remove_filters=False) -> Optional[str]:
+        """
+        Converts a dictionary to a key. Only ids from self.inputs are considered.
+
+        Parameters:
+            d (dict): Dictionary to get the key from.
+            remove_filters (bool): Option wheather the filters should be included or not.
+
+        Returns:
+            key (str): Key as string from the given dictionary.
+        """
+
+        if not isinstance(d, dict):
+            return None
+
+        new_d = copy.deepcopy(d)
+        if remove_filters:
+            for (id, _, filter) in self.inputs:
+                if filter:
+                    if id in new_d:
+                        del new_d[id]
+
+        return str(new_d)
+
+    def __call__(self, render_button=False) -> list[Component]:
+        """
+        We overwrite the get_layout method here as we use a different
+        interface compared to layout.
+        """
+
+        self.previous_inputs = {}
+        self.runs = run_handler.get_runs()
+        groups = run_handler.get_groups()
+
+        self.groups = {
+            name: GroupedRun(name, [self.runs[run_id] for run_id in run_ids])
+            for name, run_ids in groups.items()
+        }
+
+        self.all_runs = {}
+        self.all_runs.update(add_prefix_to_dict(self.runs, "run:"))
+        self.all_runs.update(add_prefix_to_dict(self.groups, "group:"))
+
+        components = [html.H1(self.name)]
+        if self.description is not None:
+            components += [html.P(self.description)]
+
+        # Register alerts
+        components += [
+            dcc.Interval(
+                id=self.get_internal_id("alert-interval"),
+                interval=1 * 500,
+                n_intervals=5,
+            ),
+            dbc.Alert(
+                id=self.get_internal_id("alert"),
+                is_open=False,
+                dismissable=True,
+                fade=True,
+            ),
+        ]
+
+        try:
+            self.check_runs_compatibility(list(self.all_runs.values()))
+        except NotMergeableError as message:
+            self.update_alert(str(message), color="danger")
+            return components
+
+        if self.activate_run_selection:
+            run_input_layout = [
+                self.__class__.get_run_input_layout(self.register_input)
+            ]
+        else:
+            run_input_layout = []
+
+        input_layout = self.__class__.get_input_layout(self.register_input)
+
+        separator_layout = []
+        if input_layout and run_input_layout:
+            separator_layout.append(html.Hr())
+
+        input_control_layout = html.Div(
+            style={} if render_button else {"display": "none"},
+            className="mt-3 clearfix",
+            children=[
+                dbc.Button(
+                    children=self.button_caption,
+                    id=self.get_internal_id("update-button"),
+                ),
+                html.Span(
+                    html.Em(id=self.get_internal_id("processing-info")),
+                    className="ms-3 align-baseline",
+                ),
+            ],
+        )
+
+        # We always have to render it because of the button.
+        # Button tells us if the page was just loaded.
+        components += [
+            html.Div(
+                id=f"{self.id}-input",
+                className="shadow-sm p-3 mb-3 bg-white rounded-lg",
+                children=run_input_layout
+                + separator_layout
+                + input_layout
+                + [input_control_layout],
+                style={}
+                if render_button or input_layout or run_input_layout
+                else {"display": "none"},
+            )
+        ]
+
+        def register_in(a, b):
+            return self.register_input(a, b, filter=True)
+
+        filter_layout = self.__class__.get_filter_layout(register_in)
+        if len(filter_layout) > 0:
+            components += [
+                html.Div(
+                    id=f"{self.id}-filter",
+                    className="shadow-sm p-3 mb-3 bg-white rounded-lg",
+                    children=filter_layout,
+                )
+            ]
+
+        output_layout = self.__class__.get_output_layout(self.register_output)
+        if output_layout:
+            components += [
+                html.Div(
+                    id=f"{self.id}-output",
+                    className="shadow-sm p-3 bg-white rounded-lg loading-container",
+                    children=output_layout,
+                    style={} if not c.get("matplotlib-mode") else {"display": "none"},
+                )
+            ]
+
+        def register_out(a, b):
+            return self.register_output(a, b, mpl=True)
+
+        output_layout = self.__class__.get_mpl_output_layout(register_out)
+        if output_layout:
+            components += [
+                html.Div(
+                    id=f"{self.id}-mpl-output",
+                    className="shadow-sm p-3 bg-white rounded-lg loading-container",
+                    children=output_layout,
+                    style={} if c.get("matplotlib-mode") else {"display": "none"},
+                )
+            ]
+
+        return components
+
+    @staticmethod
+    def get_run_input_layout(
+        register: Callable[[str, Union[str, list[str]]], str]
+    ) -> Component:
+        return html.Div(
+            [
+                dbc.Select(
+                    id=register("run_name", ["options", "value"]),
+                    placeholder="Select run ...",
+                ),
+            ]
+        )
+
+    @staticmethod
+    def load_run_inputs(
+        runs: dict[str, Run],
+        groups: dict[str, GroupedRun],
+        check_run_compatibility: Callable,
+    ) -> dict[str, Any]:
+        """
+        Set `run_names` and displays both runs and group runs if
+        they are compatible.
+        """
+
+        labels = []
+        values = []
+        disabled = []
+
+        for id, run in runs.items():
+            try:
+                values.append(f"run:{id}")
+                labels.append(id)
+                disabled.append(False)
+            except:
+                pass
+
+        added_group_label = False
+        for id, run in groups.items():
+            if check_run_compatibility(run):
+                if not added_group_label:
+                    values.append("")
+                    labels.append("Groups")
+                    disabled.append(True)
+                    added_group_label = True
+
+                values.append(f"group:{id}")
+                labels.append(id)
+                disabled.append(False)
+
+        return {
+            "run_name": {
+                "options": get_select_options(
+                    labels=labels, values=values, disabled=disabled
+                ),
+                "value": None,
+            }
+        }
+
+    def get_selected_runs(self, inputs: dict[str, Any]) -> list[AbstractRun]:
+        """
+        Parses selected runs from inputs.
+        If self.activate_run_selection is set return only selected run
+
+        Otherwise, return all possible runs
+
+        Can raise PreventUpdate() if activate_run_selection is set, but run_name not available
+        """
+
+        # Special case: If run selection is active
+        # Don't update anything if the inputs haven't changed
+        if self.activate_run_selection:
+            if inputs["run_name"]["value"] is None:
+                raise PreventUpdate()
+
+            # Update runs
+            run = run_handler.from_run_id(inputs["run_name"]["value"])
+
+            # Also:
+            # Remove `run_name` from inputs_key because
+            # we don't want the run names included.
+            _inputs = inputs.copy()
+            del _inputs["run_name"]
+
+            return [run]
+        else:
+            return list(self.all_runs.values())
+
+    def load_inputs(self) -> dict[str, Any]:
+        return {}
+
+    def load_dependency_inputs(self, previous_inputs, inputs, selected_run=None):
+        return inputs
+
+    @staticmethod
+    def get_input_layout(register) -> list[Component]:
+        return []
+
+    @staticmethod
+    def get_filter_layout(register):
+        return []
+
+    @staticmethod
+    def get_output_layout(register):
+        return []
+
+    @staticmethod
+    def get_mpl_output_layout(register):
+        return []
+
+    @staticmethod
+    def load_outputs(inputs, outputs, runs: dict[str, AbstractRun]) -> list[Component]:
+        """
+        Returns:
+            list or PreventUpdate: List of outputs (for `get_output_layout`) or PreventUpdate if
+            certain conditions are not met.
+        """
+
+        return []
+
+    @staticmethod
+    def load_mpl_outputs(inputs, outputs, runs):
+        return {}
+
+    @staticmethod
+    @abstractmethod
+    def process(run: AbstractRun, inputs):
+        pass
+
+    @staticmethod
+    def _process(
+        process: Callable[[AbstractRun, Any], None], run_cache_id: str, inputs
+    ):
+        run = run_handler.from_run_cache_id(run_cache_id)
+        return process(run, inputs)
diff --git a/deepcave/plugins/ccube.py b/deepcave/plugins/ccube.py
deleted file mode 100644
index d4c3a91c..00000000
--- a/deepcave/plugins/ccube.py
+++ /dev/null
@@ -1,212 +0,0 @@
-import numpy as np
-import pandas as pd
-from dash import dcc
-from dash import html
-import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
-import plotly.express as px
-import json
-from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.utils.logs import get_logger
-from deepcave.utils.data_structures import update_dict
-from deepcave.utils.styled_plotty import get_color
-from deepcave.utils.layout import get_slider_marks, get_select_options, get_checklist_options
-from deepcave.utils.compression import serialize, deserialize
-
-logger = get_logger(__name__)
-
-
-class CCube(DynamicPlugin):
-    def __init__(self):
-        super().__init__()
-
-    @staticmethod
-    def id():
-        return "ccube"
-
-    @staticmethod
-    def name():
-        return "Configurations Cube"
-
-    @staticmethod
-    def position():
-        return 20
-
-    @staticmethod
-    def category():
-        return "Performance Analysis"
-
-    @staticmethod
-    def activate_run_selection():
-        return True
-
-    @staticmethod
-    def get_input_layout(register):
-        return [
-            html.Div([
-                dbc.Label("Objective"),
-                dbc.Select(
-                    id=register("objective", ["options", "value"]),
-                    placeholder="Select objective ..."
-                ),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Budget"),
-                dcc.Slider(
-                    id=register("budget", ["min", "max", "marks", "value"])),
-            ]),
-        ]
-
-    @staticmethod
-    def get_filter_layout(register):
-        return [
-            html.Div([
-                dbc.Label("Number of Configurations"),
-                dcc.Slider(
-                    id=register("n_configs", ["min", "max", "marks", "value"])),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Hyperparameters"),
-                dbc.Checklist(
-                    id=register("hyperparameters", ["options", "value"])),
-            ]),
-        ]
-
-    @staticmethod
-    def load_inputs(runs):
-        return {
-            "budget": {
-                "min": 0,
-                "max": 0,
-                "marks": get_slider_marks(),
-                "value": 0
-            },
-            "n_configs": {
-                "min": 0,
-                "max": 0,
-                "marks": get_slider_marks(),
-                "value": 0
-            },
-            "objective": {
-                "options": get_select_options(),
-                "value": None
-            },
-            "hyperparameters": {
-                "options": get_checklist_options(),
-                "value": []
-            },
-        }
-
-    @staticmethod
-    def load_dependency_inputs(runs, previous_inputs, inputs):
-        run = runs[inputs["run_name"]["value"]]
-        budget_id = inputs["budget"]["value"]
-        budgets = run.get_budgets()
-        hp_names = run.configspace.get_hyperparameter_names()
-        readable_budgets = run.get_budgets(human=True)
-        configs = run.get_configs(budget=budgets[budget_id])
-        objective_names = run.get_objective_names()
-
-        objective_value = inputs["objective"]["value"]
-        if objective_value is None:
-            objective_value = objective_names[0]
-
-        new_inputs = {
-            "budget": {
-                "min": 0,
-                "max": len(readable_budgets) - 1,
-                "marks": get_slider_marks(readable_budgets),
-            },
-            "n_configs": {
-                "min": 0,
-                "max": len(configs) - 1,
-                "marks": get_slider_marks(list(range(len(configs)))),
-            },
-            "objective": {
-                "options": get_select_options(objective_names),
-                "value": objective_value
-            },
-            "hyperparameters": {
-                "options": get_select_options(hp_names),
-            },
-        }
-        update_dict(inputs, new_inputs)
-
-        # Restrict to three hyperparameters
-        selected = inputs["hyperparameters"]["value"]
-        n_selected = len(selected)
-        if n_selected > 3:
-            del selected[0]
-
-        inputs["hyperparameters"]["value"] = selected
-
-        return inputs
-
-    @staticmethod
-    def process(run, inputs):
-        objective_name = inputs["objective"]["value"]
-        budget_id = inputs["budget"]["value"]
-        budget = run.get_budget(budget_id)
-
-        df = run.get_encoded_configs(
-            objective_names=[objective_name],
-            budget=budget,
-            pandas=True
-        )
-
-        return {
-            "df": serialize(df),
-        }
-
-    @staticmethod
-    def get_output_layout(register):
-        return [
-            dcc.Graph(
-                register("graph", "figure")
-            ),
-        ]
-
-    @staticmethod
-    def load_outputs(inputs, outputs, _):
-        hp_names = inputs["hyperparameters"]["value"]
-        n_configs = inputs["n_configs"]["value"]
-
-        if n_configs == 0 or len(hp_names) == 0:
-            return [px.scatter()]
-
-        x, y, z = None, None, None
-        for i, hp_name in enumerate(hp_names):
-            if i == 0:
-                x = hp_name
-            if i == 1:
-                y = hp_name
-            if i == 2:
-                z = hp_name
-
-        run_name = inputs["run_name"]["value"]
-        df = deserialize(outputs[run_name]["df"], dtype=pd.DataFrame)
-
-        # Limit to n_configs
-        df = df.drop([str(i) for i in range(n_configs + 1, len(df))])
-
-        if x is None:
-            return [px.scatter()]
-
-        column_names = df.columns.tolist()
-        cost_name = column_names[-1]
-
-        if z is None:
-            if y is None:
-                df[""] = df[cost_name]
-                for k in df[""].keys():
-                    df[""][k] = 0
-
-                y = ""
-
-            fig = px.scatter(df, x=x, y=y, color=cost_name)
-        else:
-            fig = px.scatter_3d(df, x=x, y=y, z=z, color=cost_name)
-
-        return [fig]
diff --git a/deepcave/plugins/cost_over_time.py b/deepcave/plugins/cost_over_time.py
deleted file mode 100644
index de496de8..00000000
--- a/deepcave/plugins/cost_over_time.py
+++ /dev/null
@@ -1,239 +0,0 @@
-import numpy as np
-from dash import dcc, html
-from dash.exceptions import PreventUpdate
-import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
-from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.utils.logs import get_logger
-from deepcave.utils.styled_plotty import get_color
-from deepcave.utils.layout import get_slider_marks, get_select_options, get_checklist_options, get_radio_options
-
-
-logger = get_logger(__name__)
-
-
-class CostOverTime(DynamicPlugin):
-    def __init__(self):
-        super().__init__()
-
-    @staticmethod
-    def id():
-        return "cost_over_time"
-
-    @staticmethod
-    def name():
-        return "Cost Over Time"
-
-    @staticmethod
-    def position():
-        return 10
-
-    @staticmethod
-    def category():
-        return "Performance Analysis"
-
-    @staticmethod
-    def check_requirements(runs, _):
-        # Check if selected runs have same budgets+objectives
-        objectives = None
-        budgets = None
-
-        for _, run in runs.items():
-            if objectives is None or budgets is None:
-                objectives = run.get_objective_names()
-                budgets = run.get_budgets()
-            else:
-                if objectives != run.get_objective_names():
-                    return f"Objectives differ across the selected runs."
-                if budgets != run.get_budgets():
-                    return f"Budgets differ across the selected runs."
-
-        return True
-
-    @staticmethod
-    def get_input_layout(register):
-        return [
-            html.Div([
-                dbc.Label("Objective"),
-                dbc.Select(
-                    id=register("objective", ["options", "value"]),
-                    placeholder="Select objective ..."
-                ),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Budget"),
-                dcc.Slider(id=register(
-                           "budget", ["min", "max", "marks", "value"])),
-            ], className=""),
-        ]
-
-    @staticmethod
-    def get_filter_layout(register):
-        return [
-            html.Div([
-                dbc.Label("X-Axis"),
-                dbc.RadioItems(id=register("xaxis", ["options", "value"])),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Logarithmic"),
-                dbc.RadioItems(id=register("log", ["options", "value"])),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Show Groups"),
-                dbc.RadioItems(id=register("groups", ["options", "value"])),
-            ], className=""),
-        ]
-
-    @staticmethod
-    def load_inputs(runs):
-        # Just select the first run
-        # Since we already know the budgets+objectives
-        # are the same across the selected runs.
-        run = list(runs.values())[0]
-        readable_budgets = run.get_budgets(human=True)
-        objective_names = run.get_objective_names()
-
-        return {
-            "objective": {
-                "options": get_select_options(objective_names),
-                "value": objective_names[0]
-            },
-            "budget": {
-                "min": 0,
-                "max": len(readable_budgets) - 1,
-                "marks": get_slider_marks(readable_budgets),
-                "value": 0
-            },
-            "xaxis": {
-                "options": [
-                    {"label": "Time", "value": "times"},
-                    {"label": "Number of evaluated configurations", "value": "configs"}],
-                "value": "times"
-            },
-            "log": {
-                "options": get_radio_options(binary=True),
-                "value": True
-            },
-            "groups": {
-                "options": get_radio_options(binary=True),
-                "value": False
-            },
-        }
-
-    @staticmethod
-    def process(run, inputs):
-        budget_id = inputs["budget"]["value"]
-        budget = run.get_budget(budget_id)
-
-        costs, times, ids = run.get_trajectory(
-            objective_names=[inputs["objective"]["value"]],
-            budget=budget)
-
-        return {
-            "costs": costs,
-            "times": times,
-            "configs": ids,
-        }
-
-    @staticmethod
-    def get_output_layout(register):
-        return [
-            dcc.Graph(register("graph", "figure")),
-        ]
-
-    @staticmethod
-    def load_outputs(inputs, outputs, groups):
-        show_groups = inputs["groups"]["value"]
-        if not show_groups:
-            groups = {}
-            for run_name in outputs.keys():
-                groups[run_name] = [run_name]
-
-        traces = []
-        for group_idx, (group_name, run_names) in enumerate(groups.items()):
-            all_x = []  # All required x
-            for run_name in run_names:
-                run_x = outputs[run_name][inputs["xaxis"]["value"]]
-
-                for x in run_x:
-                    if x not in all_x:
-                        all_x.append(x)
-
-            all_x.sort()
-
-            # Now look for corresponding ys
-            all_y = []  # List of lists
-
-            for x in all_x:
-                group_y = []
-                for run_name in run_names:
-                    run_x = outputs[run_name][inputs["xaxis"]["value"]]
-                    run_y = outputs[run_name]["costs"]
-
-                    # Find closest x value
-                    idx = min(range(len(run_x)), key=lambda i: abs(run_x[i]-x))
-                    group_y.append(run_y[idx])
-
-                all_y.append(group_y)
-
-            all_x = np.array(all_x)
-            all_y = np.array(all_y)
-
-            if len(all_x) == 0 or len(all_y) == 0:
-                return PreventUpdate
-
-            y_mean = np.mean(all_y, axis=1)
-            y_std = np.std(all_y, axis=1)
-            y_upper = list(y_mean+y_std)
-            y_lower = list(y_mean-y_std)
-
-            traces.append(go.Scatter(
-                x=all_x,
-                y=y_mean,
-                name=group_name,
-                line_shape='hv',
-                line=dict(color=get_color(group_idx)),
-            ))
-
-            traces.append(go.Scatter(
-                x=all_x,
-                y=y_upper,
-                line=dict(color=get_color(group_idx, 0)),
-                line_shape='hv',
-                hoverinfo="skip",
-                showlegend=False,
-            ))
-
-            traces.append(go.Scatter(
-                x=all_x,
-                y=y_lower,
-                fill='tonexty',
-                fillcolor=get_color(group_idx, 0.2),
-                line=dict(color=get_color(group_idx, 0)),
-                line_shape='hv',
-                hoverinfo="skip",
-                showlegend=False,
-            ))
-
-        type = None
-        if inputs["log"]["value"]:
-            type = 'log'
-
-        xaxis_label = "Wallclock time [s]"
-        if inputs["xaxis"]["value"] == "configs":
-            xaxis_label = "Number of evaluated configurations"
-
-        layout = go.Layout(
-            xaxis=dict(
-                title=xaxis_label,
-                type=type
-            ),
-            yaxis=dict(
-                title=inputs["objective"]["value"],
-            ),
-        )
-
-        return [go.Figure(data=traces, layout=layout)]
diff --git a/deepcave/plugins/dynamic_plugin.py b/deepcave/plugins/dynamic_plugin.py
deleted file mode 100644
index 2542af73..00000000
--- a/deepcave/plugins/dynamic_plugin.py
+++ /dev/null
@@ -1,101 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import Dict, List, Any, Type, Union, Optional, Tuple
-import os
-import json
-import copy
-from collections import defaultdict
-
-import pandas as pd
-
-from dash.dash import no_update
-from dash.dependencies import Input, Output, State
-from dash import dcc
-import dash_bootstrap_components as dbc
-from plotly.graph_objects import Figure
-from dash import html
-from dash.development.base_component import Component
-from ConfigSpace import ConfigurationSpace
-from dash.exceptions import PreventUpdate
-
-from deepcave import app, c, rc
-from deepcave.runs.handler import handler
-from deepcave.utils.logs import get_logger
-from deepcave.plugins.plugin import Plugin
-
-
-logger = get_logger(__name__)
-
-
-class DynamicPlugin(Plugin):
-    def __init__(self):
-        super().__init__()
-
-    def register_callbacks(self):
-        super().register_callbacks()
-
-        outputs = []
-        for id, attribute, _ in self.outputs:
-            outputs.append(Output(self.get_internal_output_id(id), attribute))
-
-        inputs = [Input(self.get_internal_id("update-button"), 'n_clicks')]
-        for id, attribute, _ in self.inputs:
-            inputs.append(
-                Input(self.get_internal_input_id(id), attribute))
-
-        # Register updates from inputs
-        @app.callback(outputs, inputs)
-        def plugin_output_update(_, *inputs_list):
-            """
-            Parameters:
-                *inputs_list: Values from user.
-            """
-
-            # Map the list `inputs_list` to a dict s.t.
-            # it's easier to access them.
-            inputs = self._list_to_dict(inputs_list, input=True)
-            inputs_key = self._dict_as_key(inputs, remove_filters=True)
-
-            # Special case again
-            # Only process the selected run
-            if self.__class__.activate_run_selection():
-                if "run_name" not in inputs or inputs["run_name"]["value"] is None:
-                    raise PreventUpdate()
-
-                runs = {}
-                run_name = inputs["run_name"]["value"]
-                runs[run_name] = self.runs[run_name]
-
-                # Also:
-                # Remove `run_name` from last_inputs_key because
-                # we don't want the run names included.
-                _inputs = inputs.copy()
-                del _inputs["run_name"]
-
-                inputs_key = self._dict_as_key(_inputs, remove_filters=True)
-            else:
-                runs = self.runs
-
-            raw_outputs = {}
-            for name, run in runs.items():
-                run_outputs = rc[name].get(self.id(), inputs_key)
-                if run_outputs is None:
-                    logger.debug(f"Process {name}.")
-                    run_outputs = self.process(run, inputs)
-
-                    # Here's the thing:
-                    # We have to remove `run_name` from the inputs completely
-
-                    # Cache it
-                    rc[name].set(self.id(), inputs_key, value=run_outputs)
-                else:
-                    logger.debug(f"Found outputs from {name} in cache.")
-
-                raw_outputs[name] = run_outputs
-
-            # Cache last inputs
-            c.set("last_inputs", self.id(), value=inputs)
-
-            return self._process_raw_outputs(inputs, raw_outputs)
-
-    def __call__(self):
-        return super().__call__(False)
diff --git a/deepcave/plugins/dynamic_plugin/__init__.py b/deepcave/plugins/dynamic_plugin/__init__.py
new file mode 100644
index 00000000..8575203c
--- /dev/null
+++ b/deepcave/plugins/dynamic_plugin/__init__.py
@@ -0,0 +1,65 @@
+from abc import ABC
+
+from dash.dependencies import Input, Output
+
+from deepcave import app, c, rc
+from deepcave.plugins import Plugin
+
+
+class DynamicPlugin(Plugin, ABC):
+    def __init__(self):
+        super().__init__()
+
+    def register_callbacks(self):
+        super().register_callbacks()
+        self._callback_inputs_changed()
+
+    def _callback_inputs_changed(self):
+        outputs = []
+        for id, attribute, _ in self.outputs:
+            outputs.append(Output(self.get_internal_output_id(id), attribute))
+
+        inputs = [Input(self.get_internal_id("update-button"), "n_clicks")]
+        for id, attribute, _ in self.inputs:
+            inputs.append(Input(self.get_internal_input_id(id), attribute))
+
+        # Register updates from inputs
+        @app.callback(outputs, inputs)
+        def plugin_output_update(_, *inputs_list):
+            """
+            Parameters:
+                *inputs_list: Values from user.
+            """
+
+            # Map the list `inputs_list` to a dict s.t.
+            # it's easier to access them.
+            inputs = self._list_to_dict(inputs_list, input=True)
+            inputs_key = self._dict_as_key(inputs, remove_filters=True)
+
+            runs = self.get_selected_runs(inputs)
+
+            raw_outputs = {}
+            for run in runs:
+                run_cache = rc.get_run(run)
+                run_outputs = run_cache.get(self.id, inputs_key)
+                if run_outputs is None:
+                    self.logger.debug(f"Process {run.name}.")
+                    run_outputs = self.process(run, inputs)
+
+                    # Here's the thing:
+                    # We have to remove `run_name` from the inputs completely
+
+                    # Cache it
+                    run_cache.set(self.id, inputs_key, value=run_outputs)
+                else:
+                    self.logger.debug(f"Found outputs from {run.name} in cache.")
+
+                raw_outputs[run.name] = run_outputs
+
+            # Cache last inputs
+            c.set("last_inputs", self.id, value=inputs)
+
+            return self._process_raw_outputs(inputs, raw_outputs)
+
+    def __call__(self):
+        return super().__call__(False)
diff --git a/deepcave/plugins/dynamic_plugin/ccube.py b/deepcave/plugins/dynamic_plugin/ccube.py
new file mode 100644
index 00000000..65d1d2e0
--- /dev/null
+++ b/deepcave/plugins/dynamic_plugin/ccube.py
@@ -0,0 +1,267 @@
+import dash_bootstrap_components as dbc
+import numpy as np
+import pandas as pd
+import plotly.express as px
+from ConfigSpace.hyperparameters import CategoricalHyperparameter, Constant
+from dash import dcc, html
+
+from deepcave import run_handler
+from deepcave.plugins.dynamic_plugin import DynamicPlugin
+from deepcave.runs import AbstractRun
+from deepcave.utils.compression import deserialize, serialize
+from deepcave.utils.data_structures import update_dict
+from deepcave.utils.layout import (
+    get_checklist_options,
+    get_select_options,
+    get_slider_marks,
+)
+from deepcave.utils.logs import get_logger
+
+logger = get_logger(__name__)
+
+
+class CCube(DynamicPlugin):
+    id = "ccube"
+    name = "Configurations Cube"
+    icon = "fas fa-cube"
+
+    activate_run_selection = True
+
+    @staticmethod
+    def get_input_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Objective"),
+                    dbc.Select(
+                        id=register("objective", ["options", "value"]),
+                        placeholder="Select objective ...",
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Budget"),
+                    dcc.Slider(id=register("budget", ["min", "max", "marks", "value"])),
+                ]
+            ),
+        ]
+
+    @staticmethod
+    def get_filter_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Number of Configurations"),
+                    dcc.Slider(
+                        id=register("n_configs", ["min", "max", "marks", "value"])
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Hyperparameters"),
+                    dbc.Checklist(id=register("hyperparameters", ["options", "value"])),
+                ]
+            ),
+        ]
+
+    def load_inputs(self):
+        return {
+            "budget": {"min": 0, "max": 0, "marks": get_slider_marks(), "value": 0},
+            "n_configs": {"min": 0, "max": 0, "marks": get_slider_marks(), "value": 0},
+            "objective": {"options": get_select_options(), "value": None},
+            "hyperparameters": {"options": get_checklist_options(), "value": []},
+        }
+
+    def load_dependency_inputs(self, previous_inputs, inputs, selected_run=None):
+        budget_id = inputs["budget"]["value"]
+        budgets = selected_run.get_budgets()
+        hp_names = selected_run.configspace.get_hyperparameter_names()
+        readable_budgets = selected_run.get_budgets(human=True)
+        configs = selected_run.get_configs(budget=budgets[budget_id])
+        objective_names = selected_run.get_objective_names()
+
+        objective_value = inputs["objective"]["value"]
+        if objective_value is None:
+            objective_value = objective_names[0]
+
+        new_inputs = {
+            "budget": {
+                "min": 0,
+                "max": len(readable_budgets) - 1,
+                "marks": get_slider_marks(readable_budgets),
+            },
+            "n_configs": {
+                "min": 0,
+                "max": len(configs) - 1,
+                "marks": get_slider_marks(list(range(len(configs)))),
+            },
+            "objective": {
+                "options": get_select_options(objective_names),
+                "value": objective_value,
+            },
+            "hyperparameters": {
+                "options": get_select_options(hp_names),
+            },
+        }
+        update_dict(inputs, new_inputs)
+
+        # Restrict to three hyperparameters
+        selected = inputs["hyperparameters"]["value"]
+        n_selected = len(selected)
+        if n_selected > 3:
+            del selected[0]
+
+        inputs["hyperparameters"]["value"] = selected
+
+        return inputs
+
+    @staticmethod
+    def process(run, inputs):
+        objective_name = inputs["objective"]["value"]
+        budget_id = inputs["budget"]["value"]
+        budget = run.get_budget(budget_id)
+
+        df, df_labels = run.get_encoded_configs(
+            objective_names=[objective_name], budget=budget, pandas=True
+        )
+
+        # Now we also need to know when to use the labels and when to use the encoded data
+        show_all_labels = []
+        for hp in run.configspace.get_hyperparameters():
+            if isinstance(hp, CategoricalHyperparameter) or isinstance(hp, Constant):
+                show_all_labels.append(True)
+            else:
+                show_all_labels.append(False)
+
+        return {
+            "df": serialize(df),
+            "df_labels": serialize(df_labels),
+            "show_all_labels": show_all_labels,
+        }
+
+    @staticmethod
+    def get_output_layout(register):
+        return [
+            dcc.Graph(register("graph", "figure"), style={"height": "50vh"}),
+        ]
+
+    @staticmethod
+    def load_outputs(inputs, outputs, _):
+        hp_names = inputs["hyperparameters"]["value"]
+        n_configs = inputs["n_configs"]["value"]
+        # show_all_labels = outputs[run_name]["show_all_labels"]
+
+        if n_configs == 0 or len(hp_names) == 0:
+            return [px.scatter()]
+
+        x, y, z = None, None, None
+        for i, hp_name in enumerate(hp_names):
+            if i == 0:
+                x = hp_name
+            if i == 1:
+                y = hp_name
+            if i == 2:
+                z = hp_name
+
+        run = run_handler.from_run_id(inputs["run_name"]["value"])
+        output = outputs[run.name]
+        df = deserialize(output["df"], dtype=pd.DataFrame)
+        df_labels = deserialize(output["df_labels"], dtype=pd.DataFrame)
+
+        # Limit to n_configs
+        idx = [str(i) for i in range(n_configs + 1, len(df))]
+        df = df.drop(idx)
+        df_labels = df_labels.drop(idx)
+
+        column_names = df.columns.tolist()
+        cost_name = column_names[-1]
+
+        if x is None:
+            return [px.scatter()]
+
+        # hovertemplate = ""
+        # for name in column_names:
+        #    hovertemplate += name + ": %{df_labels[name]}<br>"
+
+        if z is None:
+            # Add another column with zeros
+            if y is None:
+                y = ""
+
+                # Do it for both dataframes
+                for frame in [df, df_labels]:
+                    frame[y] = frame[cost_name]
+                    for k in frame[y].keys():
+                        frame[y][k] = 0
+
+            fig = px.scatter(df, x=x, y=y, color=cost_name, hover_data=df_labels)
+        else:
+            fig = px.scatter_3d(
+                df, x=x, y=y, z=z, color=cost_name, hover_data=df_labels
+            )
+
+        scene = {}
+        for axis, name in zip([x, y, z], ["xaxis", "yaxis", "zaxis"]):
+            if axis is None:
+                continue
+
+            values = df[axis].values
+            labels = df_labels[axis].values
+
+            unique_values = []  # df[hp_name].unique()
+            unique_labels = []  # df_labels[hp_name].unique()
+            for value, label in zip(values, labels):
+                if value not in unique_values and label not in unique_labels:
+                    unique_values.append(value)
+                    unique_labels.append(label)
+
+            selected_values = []
+            selected_labels = []
+
+            # If we have less than 10 values, we also show them
+            # if show_all_labels[axis] or len(unique_values) < 10:
+            if len(unique_values) < 10:
+                # Make sure we don't have multiple (same) labels for the same value
+                for value, label in zip(unique_values, unique_labels):
+                    selected_values.append(value)
+                    selected_labels.append(label)
+
+            else:
+                # Add min+max values
+                for idx in [np.argmin(values), np.argmax(values)]:
+                    selected_values.append(values[idx])
+                    selected_labels.append(labels[idx])
+
+                # After we added min and max values, we want to add
+                # intermediate values too
+                min_v = np.min(values)
+                max_v = np.max(values)
+                for factor in [0.2, 0.4, 0.6, 0.8]:
+                    new_v = (factor * (max_v - min_v)) + min_v
+                    idx = np.abs(unique_values - new_v).argmin(axis=-1)
+
+                    selected_values.append(unique_values[idx])
+                    selected_labels.append(unique_labels[idx])
+
+            kwargs = {
+                name: {
+                    "tickvals": selected_values,
+                    "ticktext": selected_labels,
+                }
+            }
+            fig.update_scenes(**kwargs)
+
+            if axis is None:
+                continue
+
+            # scatter and scatter3d handle the zaxis differently
+            scene[name] = kwargs[name]
+
+        if z is None:
+            fig.update_layout(**scene)
+
+        return [fig]
diff --git a/deepcave/plugins/configurations.py b/deepcave/plugins/dynamic_plugin/configurations.py
similarity index 61%
rename from deepcave/plugins/configurations.py
rename to deepcave/plugins/dynamic_plugin/configurations.py
index 4c36a007..eeed0367 100644
--- a/deepcave/plugins/configurations.py
+++ b/deepcave/plugins/dynamic_plugin/configurations.py
@@ -1,39 +1,28 @@
-
+import dash_bootstrap_components as dbc
 import pandas as pd
+from ConfigSpace.hyperparameters import (
+    CategoricalHyperparameter,
+    Constant,
+    NormalFloatHyperparameter,
+    NormalIntegerHyperparameter,
+    OrdinalHyperparameter,
+    UniformFloatHyperparameter,
+    UniformIntegerHyperparameter,
+)
 from dash import html
-import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
+from dash.development.base_component import Component
 
+from deepcave import run_handler
 from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.utils.logs import get_logger
-from ConfigSpace.hyperparameters import UniformIntegerHyperparameter, NormalIntegerHyperparameter, UniformFloatHyperparameter, NormalFloatHyperparameter, CategoricalHyperparameter, OrdinalHyperparameter, Constant
-
-logger = get_logger(__name__)
+from deepcave.runs import AbstractRun
 
 
 class Configurations(DynamicPlugin):
-    def __init__(self):
-        super().__init__()
-
-    @staticmethod
-    def id():
-        return "configurations"
-
-    @staticmethod
-    def name():
-        return "Configurations"
+    id = "configurations"
+    name = "Configurations"
+    icon = "fas fa-sliders-h"
 
-    @staticmethod
-    def position():
-        return 5
-
-    @staticmethod
-    def category():
-        return "General"
-
-    @staticmethod
-    def activate_run_selection() -> bool:
-        return True
+    activate_run_selection = True
 
     @staticmethod
     def process(run, inputs):
@@ -49,7 +38,12 @@ def process(run, inputs):
 
             log = False
             value = None
-            if isinstance(hp, UniformIntegerHyperparameter) or isinstance(hp, NormalIntegerHyperparameter) or isinstance(hp, UniformFloatHyperparameter) or isinstance(hp, NormalFloatHyperparameter):
+            if (
+                isinstance(hp, UniformIntegerHyperparameter)
+                or isinstance(hp, NormalIntegerHyperparameter)
+                or isinstance(hp, UniformFloatHyperparameter)
+                or isinstance(hp, NormalFloatHyperparameter)
+            ):
                 value = str([hp.lower, hp.upper])
                 log = hp.log
             elif isinstance(hp, CategoricalHyperparameter):
@@ -70,10 +64,7 @@ def process(run, inputs):
         # Get best cost across all objectives, highest budget
         cost, config = run.get_min_cost()
 
-        best_config = {
-            "Hyperparameter": [],
-            "Value": []
-        }
+        best_config = {"Hyperparameter": [], "Value": []}
 
         for hp_name, value in config.items():
             best_config["Hyperparameter"].append(hp_name)
@@ -91,23 +82,23 @@ def get_output_layout(register):
             html.H3("Configuration Space"),
             html.Div(id=register("configspace", "children")),
             html.Hr(),
-
             html.H3("Best Configuration"),
             html.Div(id=register("best_config", "children")),
             html.Div(id=register("min_cost", "children")),
         ]
 
     @staticmethod
-    def load_outputs(inputs, outputs, _):
-        run_name = inputs["run_name"]["value"]
-        outputs = outputs[run_name]
+    def load_outputs(inputs, outputs, _) -> list[Component]:
+        run = run_handler.from_run_id(inputs["run_name"]["value"])
+        outputs = outputs[run.name]
 
-        def create_table(output): return dbc.Table.from_dataframe(
-            pd.DataFrame(output), striped=True, bordered=True
-        )
+        def create_table(output):
+            return dbc.Table.from_dataframe(
+                pd.DataFrame(output), striped=True, bordered=True
+            )
 
         return [
             create_table(outputs["configspace"]),
             create_table(outputs["best_config"]),
-            f"With normalized cost: {outputs['min_cost']}"
+            f"With normalized cost: {outputs['min_cost']}",
         ]
diff --git a/deepcave/plugins/dynamic_plugin/cost_over_time.py b/deepcave/plugins/dynamic_plugin/cost_over_time.py
new file mode 100644
index 00000000..5daffd44
--- /dev/null
+++ b/deepcave/plugins/dynamic_plugin/cost_over_time.py
@@ -0,0 +1,198 @@
+from typing import Union, List
+
+import dash_bootstrap_components as dbc
+import numpy as np
+import plotly.graph_objs as go
+from dash import dcc, html
+from dash.development.base_component import Component
+from dash.exceptions import PreventUpdate
+
+from deepcave.plugins.dynamic_plugin import DynamicPlugin
+from deepcave.runs import AbstractRun, NotMergeableError, check_equality
+from deepcave.utils.layout import (
+    get_radio_options,
+    get_select_options,
+    get_slider_marks,
+)
+from deepcave.utils.styled_plotty import get_color
+
+
+class CostOverTime(DynamicPlugin):
+    id = "cost_over_time"
+    name = "Cost Over Time"
+    icon = "fas fa-chart-line"
+
+    def check_runs_compatibility(self, runs: List[AbstractRun]) -> None:
+        check_equality(runs, objectives=True, budgets=True)
+
+        # Set some attributes here
+        run = runs[0]
+        self.readable_budgets = run.get_budgets(human=True)
+        self.objective_names = run.get_objective_names()
+
+    @staticmethod
+    def get_input_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Objective"),
+                    dbc.Select(
+                        id=register("objective", ["options", "value"]),
+                        placeholder="Select objective ...",
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Budget"),
+                    dcc.Slider(id=register("budget", ["min", "max", "marks", "value"])),
+                ],
+                className="",
+            ),
+        ]
+
+    @staticmethod
+    def get_filter_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("X-Axis"),
+                    dbc.RadioItems(id=register("xaxis", ["options", "value"])),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Display ..."),
+                    dbc.RadioItems(id=register("display", ["options", "value"])),
+                ],
+                className="",
+            ),
+        ]
+
+    def load_inputs(self):
+        display_options = ["Runs", "Groups"]
+
+        return {
+            "objective": {
+                "options": get_select_options(self.objective_names),
+                "value": self.objective_names[0],
+            },
+            "budget": {
+                "min": 0,
+                "max": len(self.readable_budgets) - 1,
+                "marks": get_slider_marks(self.readable_budgets),
+                "value": 0,
+            },
+            "xaxis": {
+                "options": [
+                    {"label": "Time", "value": "times"},
+                    {"label": "Time (logarithmic)", "value": "times_log"},
+                    {"label": "Number of evaluated configurations", "value": "configs"},
+                ],
+                "value": "times",
+            },
+            "display": {
+                "options": get_radio_options(display_options),
+                "value": display_options[0],
+            },
+        }
+
+    @staticmethod
+    def process(run, inputs) -> dict[str, list[Union[float, str]]]:
+        budget_id = inputs["budget"]["value"]
+        budget = run.get_budget(budget_id)
+
+        times, costs_mean, costs_std, ids = run.get_trajectory(
+            objective_names=[inputs["objective"]["value"]], budget=budget
+        )
+
+        return {
+            "times": times,
+            "costs_mean": costs_mean,
+            "costs_std": costs_std,
+            "ids": ids,
+        }
+
+    @staticmethod
+    def get_output_layout(register):
+        return [
+            dcc.Graph(register("graph", "figure")),
+        ]
+
+    @staticmethod
+    def load_outputs(inputs, outputs, runs: dict[str, AbstractRun]) -> list[Component]:
+
+        traces = []
+        for idx, (run_name, run) in enumerate(runs.items()):
+            x = outputs[run.name]["times"]
+            if inputs["xaxis"]["value"] == "configs":
+                x = outputs[run.name]["ids"]
+
+            if inputs["display"]["value"] == "Runs":
+                if run.prefix == "group":
+                    continue
+            elif inputs["display"]["value"] == "Groups":
+                # Prefix could be not only run but also the name of the converter
+                if run.prefix != "group":
+                    continue
+            else:
+                raise RuntimeError("Unknown display option")
+
+            y = np.array(outputs[run.name]["costs_mean"])
+            y_err = np.array(outputs[run.name]["costs_std"])
+            y_upper = list(y + y_err)
+            y_lower = list(y - y_err)
+            y = list(y)
+
+            traces.append(
+                go.Scatter(
+                    x=x,
+                    y=y,
+                    name=run_name,
+                    line_shape="hv",
+                    line=dict(color=get_color(idx)),
+                )
+            )
+
+            traces.append(
+                go.Scatter(
+                    x=x,
+                    y=y_upper,
+                    line=dict(color=get_color(idx, 0)),
+                    line_shape="hv",
+                    hoverinfo="skip",
+                    showlegend=False,
+                )
+            )
+
+            traces.append(
+                go.Scatter(
+                    x=x,
+                    y=y_lower,
+                    fill="tonexty",
+                    fillcolor=get_color(idx, 0.2),
+                    line=dict(color=get_color(idx, 0)),
+                    line_shape="hv",
+                    hoverinfo="skip",
+                    showlegend=False,
+                )
+            )
+
+        type = None
+        if inputs["xaxis"]["value"] == "times_log":
+            type = "log"
+
+        xaxis_label = "Wallclock time [s]"
+        if inputs["xaxis"]["value"] == "configs":
+            xaxis_label = "Number of evaluated configurations"
+
+        layout = go.Layout(
+            xaxis=dict(title=xaxis_label, type=type),
+            yaxis=dict(
+                title=inputs["objective"]["value"],
+            ),
+        )
+
+        return [go.Figure(data=traces, layout=layout)]
diff --git a/deepcave/plugins/overview.py b/deepcave/plugins/dynamic_plugin/overview.py
similarity index 65%
rename from deepcave/plugins/overview.py
rename to deepcave/plugins/dynamic_plugin/overview.py
index c6b5e98d..981a9460 100644
--- a/deepcave/plugins/overview.py
+++ b/deepcave/plugins/dynamic_plugin/overview.py
@@ -1,56 +1,24 @@
-from typing import Dict, Type, Any
-
-from dash import dcc
-from dash import html
 import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
-
 import pandas as pd
-import numpy as np
+from dash import html
 
-from deepcave import app
-from deepcave.plugins.static_plugin import StaticPlugin
+from deepcave import run_handler
 from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.utils.logs import get_logger
-from deepcave.runs.run import Status
-
-from deepcave.evaluators.fanova import fANOVA as _fANOVA
-
-
-logger = get_logger(__name__)
+from deepcave.runs import AbstractRun, Status
+from deepcave.runs.grouped_run import NotMergeableError
 
 
 class Overview(DynamicPlugin):
-    def __init__(self):
-        super().__init__()
-
-    @staticmethod
-    def id():
-        return "overview"
-
-    @staticmethod
-    def name():
-        return "Overview"
+    id = "overview"
+    name = "Overview"
+    icon = "fas fa-search"
 
-    @staticmethod
-    def position():
-        return 1
-
-    @staticmethod
-    def category():
-        return "General"
-
-    @staticmethod
-    def activate_run_selection() -> bool:
-        return True
+    activate_run_selection = True
 
     @staticmethod
     def process(run, inputs):
         # Meta
-        meta = {
-            "Attribute": [],
-            "Value": []
-        }
+        meta = {"Attribute": [], "Value": []}
         for k, v in run.get_meta().items():
             if k == "objectives":
                 continue
@@ -62,11 +30,7 @@ def process(run, inputs):
             meta["Value"].append(str(v))
 
         # Objectives
-        objectives = {
-            "Name": [],
-            "Lower Bound": [],
-            "Upper Bound": []
-        }
+        objectives = {"Name": [], "Lower Bound": [], "Upper Bound": []}
         for objective in run.get_objectives():
             objectives["Name"].append(objective["name"])
             objectives["Lower Bound"].append(objective["lower"])
@@ -83,13 +47,10 @@ def process(run, inputs):
                 for s in Status:
                     stats[budget][s.name] = 0
 
-        for trial in run.history:
+        for trial in run.get_trials():
             stats[trial.budget][trial.status.name] += 1
 
-        statistics = {
-            "Budget": budgets,
-            "Total": []
-        }
+        statistics = {"Budget": budgets, "Total": []}
         for s in Status:
             statistics[s.name] = []
 
@@ -121,24 +82,22 @@ def get_output_layout(register):
             html.H3("Meta"),
             html.Div(id=register("meta", "children")),
             html.Hr(),
-
             html.H3("Objectives"),
             html.Div(id=register("objectives", "children")),
             html.Hr(),
-
             html.H3("Statistics"),
             html.Div(id=register("statistics", "children")),
-            html.Hr(),
         ]
 
     @staticmethod
     def load_outputs(inputs, outputs, _):
-        run_name = inputs["run_name"]["value"]
-        outputs = outputs[run_name]
+        run = run_handler.from_run_id(inputs["run_name"]["value"])
+        outputs = outputs[run.name]
 
-        def create_table(output): return dbc.Table.from_dataframe(
-            pd.DataFrame(output), striped=True, bordered=True
-        )
+        def create_table(output):
+            return dbc.Table.from_dataframe(
+                pd.DataFrame(output), striped=True, bordered=True
+            )
 
         return [
             create_table(outputs["meta"]),
diff --git a/deepcave/plugins/dynamic_plugin/parallel_coordinates.py b/deepcave/plugins/dynamic_plugin/parallel_coordinates.py
new file mode 100644
index 00000000..a2d4ffdb
--- /dev/null
+++ b/deepcave/plugins/dynamic_plugin/parallel_coordinates.py
@@ -0,0 +1,220 @@
+from typing import Optional
+
+from collections import defaultdict
+
+import dash_bootstrap_components as dbc
+import numpy as np
+import pandas as pd
+import plotly.express as px
+import plotly.graph_objs as go
+from ConfigSpace.hyperparameters import CategoricalHyperparameter, Constant
+from dash import dcc, html
+
+from deepcave import run_handler
+from deepcave.plugins.dynamic_plugin import DynamicPlugin
+from deepcave.utils.compression import deserialize, serialize
+from deepcave.utils.data_structures import update_dict
+from deepcave.utils.layout import (
+    get_checklist_options,
+    get_select_options,
+    get_slider_marks,
+)
+from deepcave.utils.logs import get_logger
+
+logger = get_logger(__name__)
+
+
+class ParallelCoordinates(DynamicPlugin):
+    id = "parallel_coordinates"
+    name = "Parallel Coordinates"
+    description = """
+        This type of visualisation is used for plotting multivariate, numerical data. Parallel
+        Coordinates Plots are ideal for comparing many variables together and
+        seeing the relationships between them. For example, if you had to compare an array
+        of products with the same attributes (comparing computer or cars specs across
+        different models).
+    """
+    icon = "far fa-map"
+    activate_run_selection = True
+
+    @staticmethod
+    def get_input_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Objective"),
+                    dbc.Select(
+                        id=register("objective", ["options", "value"]),
+                        placeholder="Select objective ...",
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Budget"),
+                    dcc.Slider(id=register("budget", ["min", "max", "marks", "value"])),
+                ]
+            ),
+        ]
+
+    @staticmethod
+    def get_filter_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Hyperparameters"),
+                    dbc.Checklist(id=register("hyperparameters", ["options", "value"])),
+                ]
+            ),
+        ]
+
+    def load_inputs(self):
+        return {
+            "objective": {"options": get_select_options(), "value": None},
+            "budget": {"min": 0, "max": 0, "marks": get_slider_marks(), "value": 0},
+            "hyperparameters": {"options": get_checklist_options(), "value": []},
+        }
+
+    def load_dependency_inputs(self, previous_inputs, inputs, selected_run):
+        hp_names = selected_run.configspace.get_hyperparameter_names()
+        readable_budgets = selected_run.get_budgets(human=True)
+        objective_names = selected_run.get_objective_names()
+
+        objective_value = inputs["objective"]["value"]
+        if objective_value is None:
+            objective_value = objective_names[0]
+
+        new_inputs = {
+            "objective": {
+                "options": get_select_options(objective_names),
+                "value": objective_value,
+            },
+            "budget": {
+                "min": 0,
+                "max": len(readable_budgets) - 1,
+                "marks": get_slider_marks(readable_budgets),
+            },
+            "hyperparameters": {
+                "options": get_select_options(hp_names),
+            },
+        }
+        update_dict(inputs, new_inputs)
+
+        return inputs
+
+    @staticmethod
+    def process(run, inputs):
+        objective_name = inputs["objective"]["value"]
+        budget_id = inputs["budget"]["value"]
+        budget = run.get_budget(budget_id)
+
+        df, df_labels = run.get_encoded_configs(
+            objective_names=[objective_name], budget=budget, pandas=True
+        )
+
+        # Now we also need to know when to use the labels and when to use the encoded data
+        show_all_labels = []
+        for hp in run.configspace.get_hyperparameters():
+            if isinstance(hp, CategoricalHyperparameter) or isinstance(hp, Constant):
+                show_all_labels.append(True)
+            else:
+                show_all_labels.append(False)
+
+        return {
+            "df": serialize(df),
+            "df_labels": serialize(df_labels),
+            "show_all_labels": show_all_labels,
+        }
+
+    @staticmethod
+    def get_output_layout(register):
+        return [
+            dcc.Graph(register("graph", "figure")),
+        ]
+
+    @staticmethod
+    def load_outputs(inputs, outputs, _):
+        hp_names = inputs["hyperparameters"]["value"]
+        run = run_handler.from_run_id(inputs["run_name"]["value"])
+        outputs = outputs[run.name]
+
+        show_all_labels = outputs["show_all_labels"]
+
+        df = outputs["df"]
+        df = deserialize(df, dtype=pd.DataFrame)
+        df_labels = outputs["df_labels"]
+        df_labels = deserialize(df_labels, dtype=pd.DataFrame)
+
+        # Dummy data to understand the structure
+        # data = {
+        #     "hp1": {
+        #         "values": [0, 1, 2],
+        #         "label": "HP1",
+        #     },
+        #     "hp2": {
+        #         "values": [0, 4, 2],
+        #         "label": "HP2",
+        #     },
+        # }
+
+        data = defaultdict(dict)
+        for hp_name, show_all in zip(hp_names, show_all_labels):
+            values = df[hp_name].values
+            labels = df_labels[hp_name].values
+
+            unique_values = []  # df[hp_name].unique()
+            unique_labels = []  # df_labels[hp_name].unique()
+            for value, label in zip(values, labels):
+                if value not in unique_values and label not in unique_labels:
+                    unique_values.append(value)
+                    unique_labels.append(label)
+
+            data[hp_name]["values"] = values
+            data[hp_name]["label"] = hp_name
+
+            selected_values = []
+            selected_labels = []
+
+            # If we have less than 10 values, we also show them
+            if show_all or len(unique_values) < 10:
+                # Make sure we don't have multiple (same) labels for the same value
+                for value, label in zip(unique_values, unique_labels):
+                    selected_values.append(value)
+                    selected_labels.append(label)
+
+            else:
+                # Add min+max values
+                for idx in [np.argmin(values), np.argmax(values)]:
+                    selected_values.append(values[idx])
+                    selected_labels.append(labels[idx])
+
+                # After we added min and max values, we want to add
+                # intermediate values too
+                min_v = np.min(values)
+                max_v = np.max(values)
+                for factor in [0.2, 0.4, 0.6, 0.8]:
+                    new_v = (factor * (max_v - min_v)) + min_v
+                    idx = np.abs(unique_values - new_v).argmin(axis=-1)
+
+                    selected_values.append(unique_values[idx])
+                    selected_labels.append(unique_labels[idx])
+
+            data[hp_name]["tickvals"] = selected_values
+            data[hp_name]["ticktext"] = selected_labels
+
+        objective = inputs["objective"]["value"]
+        data[objective]["values"] = df[objective].values
+        data[objective]["label"] = objective
+
+        fig = go.Figure(
+            data=go.Parcoords(
+                line=dict(
+                    color=data[objective]["values"],
+                    showscale=True,
+                ),
+                dimensions=list([d for d in data.values()]),
+            )
+        )
+
+        return [fig]
diff --git a/deepcave/plugins/ice.py b/deepcave/plugins/ice.py
deleted file mode 100644
index 288df9c0..00000000
--- a/deepcave/plugins/ice.py
+++ /dev/null
@@ -1,195 +0,0 @@
-import numpy as np
-from dash import dcc
-from dash import html
-from dash.exceptions import PreventUpdate
-import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
-import plotly.express as px
-import json
-from deepcave.plugins.dynamic_plugin import DynamicPlugin
-from deepcave.plugins.static_plugin import StaticPlugin
-from deepcave.utils.logs import get_logger
-from deepcave.utils.data_structures import update_dict
-from deepcave.utils.styled_plotty import get_color
-from deepcave.utils.layout import get_slider_marks, get_select_options, get_checklist_options, get_radio_options
-from deepcave.utils.compression import serialize, deserialize
-from deepcave.evaluators.ice import ICE as ICEEvaluator
-
-logger = get_logger(__name__)
-
-
-class ICE(StaticPlugin):
-    def __init__(self):
-        super().__init__()
-
-    @staticmethod
-    def id():
-        return "ice"
-
-    @staticmethod
-    def name():
-        return "Individual Conditional Expectation"
-
-    @staticmethod
-    def position():
-        return 30
-
-    @staticmethod
-    def category():
-        return "Performance Analysis"
-
-    @staticmethod
-    def activate_run_selection():
-        return True
-
-    @staticmethod
-    def get_input_layout(register):
-        return [
-            html.Div([
-                dbc.Label("Objective"),
-                dbc.Select(
-                    id=register("objective", ["options", "value"]),
-                    placeholder="Select objective ..."
-                ),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Budget"),
-                dcc.Slider(
-                    id=register("budget", ["min", "max", "marks", "value"])),
-            ]),
-        ]
-
-    @staticmethod
-    def get_filter_layout(register):
-        return [
-            html.Div([
-                dbc.Label("Hyperparameters"),
-                dbc.RadioItems(
-                    id=register("hyperparameters", ["options", "value"])),
-            ]),
-        ]
-
-    @staticmethod
-    def load_inputs(runs):
-        return {
-            "budget": {
-                "min": 0,
-                "max": 0,
-                "marks": get_slider_marks(),
-                "value": 0
-            },
-            "objective": {
-                "options": get_select_options(),
-                "value": None
-            },
-            "hyperparameters": {
-                "options": get_radio_options(),
-                "value": None
-            },
-        }
-
-    @staticmethod
-    def load_dependency_inputs(runs, previous_inputs, inputs):
-        run = runs[inputs["run_name"]["value"]]
-        budget_id = inputs["budget"]["value"]
-        budgets = run.get_budgets()
-        hp_names = run.configspace.get_hyperparameter_names()
-        hp_idx = [run.configspace.get_idx_by_hyperparameter_name(
-            hp_name) for hp_name in hp_names]
-        readable_budgets = run.get_budgets(human=True)
-        configs = run.get_configs(budget=budgets[budget_id])
-        objective_names = run.get_objective_names()
-
-        objective_value = inputs["objective"]["value"]
-        if objective_value is None:
-            objective_value = objective_names[0]
-
-        new_inputs = {
-            "budget": {
-                "min": 0,
-                "max": len(readable_budgets) - 1,
-                "marks": get_slider_marks(readable_budgets),
-            },
-            "objective": {
-                "options": get_select_options(objective_names),
-                "value": objective_value
-            },
-            "hyperparameters": {
-                "options": get_radio_options(hp_names, hp_idx),
-            },
-        }
-        update_dict(inputs, new_inputs)
-
-        return inputs
-
-    @staticmethod
-    def process(run, inputs):
-        objective_name = inputs["objective"]["value"]
-        budget_id = inputs["budget"]["value"]
-        budget = run.get_budget(budget_id)
-
-        X, Y = run.get_encoded_configs(
-            objective_names=[objective_name],
-            budget=budget,
-        )
-
-        evaluator = ICEEvaluator()
-        evaluator.fit(run.configspace, X, Y)
-
-        return {
-            "data": serialize(evaluator.get_data())
-        }
-
-    @staticmethod
-    def get_output_layout(register):
-        return [
-            dcc.Graph(register("graph", "figure")),
-        ]
-
-    @staticmethod
-    def load_outputs(inputs, outputs, _):
-        s = inputs["hyperparameters"]["value"]
-
-        if s is None:
-            return PreventUpdate
-
-        run_name = inputs["run_name"]["value"]
-        hp_name = inputs["hyperparameters"]["options"][s]["label"]
-        run_output = outputs[run_name]
-
-        data = deserialize(run_output["data"], dtype=np.ndarray)
-        evaluator = ICEEvaluator(data)
-        all_x, all_y = evaluator.get_ice_data(s)
-
-        traces = []
-        for x, y in zip(all_x, all_y):
-            traces.append(
-                go.Scatter(
-                    x=x,
-                    y=y,
-                    showlegend=False,
-                    line_color=get_color(0, alpha=0.05),
-                    hoverinfo='skip'
-                ))
-
-        x, y = evaluator.get_pdp_data(s)
-        traces.append(
-            go.Scatter(
-                x=x,
-                y=y,
-                showlegend=False,
-                line_color=get_color(0, alpha=1)
-            )
-        )
-
-        layout = go.Layout(
-            xaxis=dict(
-                title=hp_name,
-            ),
-            yaxis=dict(
-                title=inputs["objective"]["value"],
-            ),
-        )
-
-        return [go.Figure(data=traces, layout=layout)]
diff --git a/deepcave/plugins/pending/budget_correlation.py b/deepcave/plugins/pending/budget_correlation.py
new file mode 100644
index 00000000..e20e4e43
--- /dev/null
+++ b/deepcave/plugins/pending/budget_correlation.py
@@ -0,0 +1,76 @@
+from typing import Any, Dict, Type
+
+from itertools import combinations_with_replacement
+
+import dash_bootstrap_components as dbc
+import numpy as np
+import pandas as pd
+import plotly.express as px
+from dash import dcc, html
+from scipy.stats import spearmanr
+
+from deepcave.plugins.plugin import Plugin
+from deepcave.utils.logs import get_logger
+from deepcave.utils.styled_plot import plt
+
+logger = get_logger(__name__)
+
+'''
+class BudgetCorrelation(Plugin):
+    @staticmethod
+    def id():
+        return "budget_correlation"
+
+    @staticmethod
+    def name():
+        return "Budget Correlation"
+
+    def _get_input_layout(self):
+        return [
+            dbc.Input(id=self.register_input("objective")),
+            dbc.Select(id=self.register_input("fidelity", attributes=["value", "options"])),
+        ]
+    
+    def _get_output_layout(self):
+        return [
+            html.Div(id=self.register_output("test", func=self.plot_figure)),
+            dbc.Input(id=self.register_output("output")),
+            dbc.Input(id=self.register_output("output2"))
+        ]
+
+    def plot_figure(self, data):
+        fig = plt.figure()
+        plt.plot([i for i in range(int(data["yay"]))])
+        plt.ylabel(data["blub"])
+        plt.xlabel(data["blub"])
+
+        return [
+            display_figure(fig)
+        ]
+
+    def _load_input(self, run, **inputs):
+        return {
+            "objective": {
+                "value": "hi"
+            },
+            "fidelity": {
+                "value": '2',
+                "options": [{"label": i, "value": i} for i in range(5)]
+            }
+        }
+
+    def _load_output(self, run, **inputs):
+        """
+        It's important to return only serializable data.
+        Layouts should be specified within _get_output_layout.
+        """
+
+        objective = inputs["objective"]["value"]
+        fidelity = inputs["fidelity"]["value"]
+
+        return {
+            "test": {"blub": objective, "yay": fidelity},
+            "output": objective,
+            "output2": fidelity
+        }
+'''
diff --git a/deepcave/plugins/pending/configspace.py b/deepcave/plugins/pending/configspace.py
new file mode 100644
index 00000000..5719e125
--- /dev/null
+++ b/deepcave/plugins/pending/configspace.py
@@ -0,0 +1,108 @@
+from typing import Any, Dict, Type
+
+import dash_bootstrap_components as dbc
+import numpy as np
+import pandas as pd
+import plotly.graph_objs as go
+from dash import dcc, html
+
+from deepcave import app
+from deepcave.evaluators.fanova import fANOVA as _fANOVA
+from deepcave.plugins.dynamic_plugin import DynamicPlugin
+from deepcave.plugins.static_plugin import StaticPlugin
+from deepcave.runs.run import Status
+from deepcave.utils.logs import get_logger
+
+logger = get_logger(__name__)
+
+"""
+class Configspace(DynamicPlugin):
+    @staticmethod
+    def id():
+        return "configspace"
+
+    @staticmethod
+    def name():
+        return "Configspace"
+
+    @staticmethod
+    def position():
+        return 2
+
+    @staticmethod
+    def category():
+        return "Meta-Data Analysis"
+
+    @staticmethod
+    def description():
+        return "especially lists all found configurations and which fidelities were used."
+
+    @staticmethod
+    def debug():
+        return False
+
+    def get_input_layout(self):
+        return []
+
+    def get_filter_layout(self):
+        return []
+
+    @staticmethod
+    def process(run, inputs):
+
+        # Config id | Fidelity #1 | Fidelity #2 | ...
+        all_config_ids = []
+        config_ids = {}
+        for trial in run.history:
+            if trial.status != Status.SUCCESS:
+                continue
+
+            if trial.config_id not in all_config_ids:
+                all_config_ids.append(trial.config_id)
+
+            if trial.budget not in config_ids:
+                config_ids[trial.budget] = []
+
+            if trial.config_id not in config_ids[trial.budget]:
+                config_ids[trial.budget].append(trial.config_id)
+
+        results = {}
+        for config_id in all_config_ids:
+            results[config_id] = []
+            for fidelity in run.get_budgets():
+                if config_id in config_ids[fidelity]:
+                    results[config_id].append("YES")
+                else:
+                    results[config_id].append("")
+
+        return {
+            "fidelities": run.get_budgets(),
+            "data": results
+        }
+
+    def get_output_layout(self):
+        return [
+            html.Div(id=self.register_output("output", "children"))
+        ]
+
+    def load_outputs(self, filters, raw_outputs):
+        table_header = [
+            html.Thead(html.Tr([
+                html.Th("Config ID"),
+                *[html.Th(fidelity) for fidelity in raw_outputs["fidelities"]]
+            ]))
+        ]
+
+        rows = []
+        for config_id, values in raw_outputs["data"].items():
+            fidelity_cols = []
+            for value in values:
+                fidelity_cols.append(html.Td(value))
+
+            rows.append(html.Tr([html.Td(config_id), *fidelity_cols]))
+
+        table_body = [html.Tbody(rows)]
+        table = dbc.Table(table_header + table_body, bordered=True)
+
+        return table
+"""
diff --git a/deepcave/plugins/pending/ice.py b/deepcave/plugins/pending/ice.py
new file mode 100644
index 00000000..f786b509
--- /dev/null
+++ b/deepcave/plugins/pending/ice.py
@@ -0,0 +1,201 @@
+from typing import List
+import dash_bootstrap_components as dbc
+import numpy as np
+import plotly.graph_objs as go
+from dash import dcc, html
+from dash.exceptions import PreventUpdate
+
+from deepcave.evaluators.ice import ICE as ICEEvaluator
+from deepcave.plugins.static_plugin import StaticPlugin
+from deepcave.runs import AbstractRun, check_equality
+from deepcave.utils.compression import deserialize, serialize
+from deepcave.utils.layout import (
+    get_radio_options,
+    get_select_options,
+    get_slider_marks,
+)
+from deepcave.utils.styled_plotty import get_color
+
+"""
+class ICE(StaticPlugin):
+    id = "ice"
+    name = "Individual Conditional Expectation"
+    icon = "fas fa-grip-lines"
+    activate_run_selection = True
+
+    @staticmethod
+    def get_input_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Objective"),
+                    dbc.Select(
+                        id=register("objective", ["options", "value"]),
+                        placeholder="Select objective ...",
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Budget"),
+                    dcc.Slider(id=register("budget", ["min", "max", "marks", "value"])),
+                ]
+            ),
+        ]
+
+    @staticmethod
+    def get_filter_layout(register):
+        return [
+            html.Div(
+                [
+                    dbc.Label("Hyperparameters"),
+                    dbc.RadioItems(
+                        id=register("hyperparameters", ["options", "value"])
+                    ),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Show Confidence Bands"),
+                    dbc.RadioItems(
+                        id=register("confidence_bands", ["options", "value"])
+                    ),
+                ],
+                className="",
+            ),
+        ]
+
+    def load_inputs(self):
+        hp_names = run.configspace.get_hyperparameter_names()
+        hp_idx = [
+            run.configspace.get_idx_by_hyperparameter_name(hp_name)
+            for hp_name in hp_names
+        ]
+        readable_budgets = run.get_budgets(human=True)
+        objective_names = run.get_objective_names()
+
+        budget_marks = get_slider_marks(readable_budgets)
+        objective_options = get_select_options(objective_names)
+        hp_options = get_radio_options(hp_names, hp_idx)
+        ci_options = get_radio_options(binary=True)
+
+        return {
+            "budget": {
+                "min": 0,
+                "max": len(readable_budgets) - 1,
+                "marks": budget_marks,
+                "value": 0,
+            },
+            "objective": {
+                "options": objective_options,
+                "value": objective_options[0]["value"],
+            },
+            "hyperparameters": {"options": hp_options, "value": hp_options[0]["value"]},
+            "confidence_bands": {
+                "options": ci_options,
+                "value": ci_options[0]["value"],
+            },
+        }
+
+    @staticmethod
+    def process(run, inputs):
+        objective_name = inputs["objective"]["value"]
+        budget_id = inputs["budget"]["value"]
+        budget = run.get_budget(budget_id)
+
+        X, Y = run.get_encoded_configs(
+            objective_names=[objective_name],
+            budget=budget,
+        )
+
+        evaluator = ICEEvaluator()
+        evaluator.fit(run.configspace, X, Y)
+
+        return {"data": serialize(evaluator.get_data())}
+
+    @staticmethod
+    def get_output_layout(register):
+        return [
+            dcc.Graph(register("graph-mean", "figure")),
+        ]
+
+    @staticmethod
+    def load_outputs(inputs, outputs, _):
+        s = inputs["hyperparameters"]["value"]
+
+        if s is None:
+            return PreventUpdate
+
+        hp_name = inputs["hyperparameters"]["options"][s]["label"]
+
+        traces = []
+        for idx, (run_name, run_outputs) in enumerate(outputs.items()):
+            data = deserialize(run_outputs["data"], dtype=np.ndarray)
+            evaluator = ICEEvaluator(data)
+
+            all_x, all_y = evaluator.get_ice_data(s, variance_based=False)
+            x, y, y_std = evaluator.get_pdp_data(s, variance_based=False)
+
+            y_upper = list(y + y_std)
+            y_lower = list(y - y_std)
+            y_hat = np.mean(y, axis=0)
+
+            traces.append(
+                go.Scatter(
+                    x=x,
+                    y=y,
+                    showlegend=True,
+                    name=f"{run_name} ({np.round(y_hat, 3)})",
+                    line_color=get_color(idx, alpha=1),
+                )
+            )
+
+            if not inputs["confidence_bands"]["value"]:
+                for x, y in zip(all_x, all_y):
+                    traces.append(
+                        go.Scatter(
+                            x=x,
+                            y=y,
+                            showlegend=False,
+                            line_color=get_color(idx, alpha=0.05),
+                            hoverinfo="skip",
+                        )
+                    )
+            else:
+                traces.append(
+                    go.Scatter(
+                        x=x,
+                        y=y_upper,
+                        line=dict(color=get_color(idx, 0)),
+                        # line_shape='hv',
+                        hoverinfo="skip",
+                        showlegend=False,
+                    )
+                )
+
+                traces.append(
+                    go.Scatter(
+                        x=x,
+                        y=y_lower,
+                        fill="tonexty",
+                        fillcolor=get_color(idx, 0.2),
+                        line=dict(color=get_color(idx, 0)),
+                        # line_shape='hv',
+                        hoverinfo="skip",
+                        showlegend=False,
+                    )
+                )
+
+        layout = go.Layout(
+            xaxis=dict(
+                title=hp_name,
+            ),
+            yaxis=dict(
+                title=inputs["objective"]["value"],
+            ),
+        )
+
+        return go.Figure(data=traces, layout=layout)
+"""
diff --git a/deepcave/plugins/pending/template.py b/deepcave/plugins/pending/template.py
new file mode 100644
index 00000000..aeb9af45
--- /dev/null
+++ b/deepcave/plugins/pending/template.py
@@ -0,0 +1,87 @@
+import dash_bootstrap_components as dbc
+import numpy as np
+import plotly.graph_objs as go
+from dash import dcc
+
+from deepcave.plugins.dynamic_plugin import DynamicPlugin
+from deepcave.utils.logs import get_logger
+from deepcave.utils.styled_plotty import get_color
+
+logger = get_logger(__name__)
+
+"""
+class Template(DynamicPlugin):
+    def __init__(self):
+        super().__init__()
+
+    @staticmethod
+    def id():
+        return "template"
+
+    @staticmethod
+    def name():
+        return "Template"
+
+    @staticmethod
+    def position():
+        return 1
+
+    @staticmethod
+    def category():
+        return "Examples"
+
+    @staticmethod
+    def get_input_layout(register):
+        return [
+            dbc.Label("Test input"),
+            dbc.Input(id=register("test-input", "value"))
+        ]
+
+    @staticmethod
+    def get_filter_layout(register):
+        return [
+            dbc.Label("Test radio items"),
+            dbc.RadioItems(id=register("test-radio", ["options", "value"]))
+        ]
+
+    @staticmethod
+    def load_inputs(runs):
+        return {
+            "test-radio": {
+                "options": [{"label": "Yes", "value": True}, {"label": "No", "value": False}],
+                "value": True
+            },
+        }
+
+    @staticmethod
+    def process(run, inputs):
+        return {
+            "x": [1, 2, 3],
+            "y": [4, 2, 6]
+        }
+
+    @staticmethod
+    def get_output_layout(register):
+        return [
+            dbc.Input(id=register("test-output", "value")),
+            dcc.Graph(register("graph", "figure")),
+        ]
+
+    @staticmethod
+    def load_outputs(inputs, outputs, groups):
+        traces = []
+
+        for run_name, run_outputs in outputs.items():
+            trace = go.Scatter(
+                x=run_outputs["x"],
+                y=run_outputs["y"],
+                name=run_name,
+            )
+
+            traces.append(trace)
+
+        return [
+            inputs["test-input"]["value"],
+            go.Figure(data=traces)
+        ]
+"""
diff --git a/deepcave/plugins/plugin.py b/deepcave/plugins/plugin.py
deleted file mode 100644
index 9f8d9e7f..00000000
--- a/deepcave/plugins/plugin.py
+++ /dev/null
@@ -1,563 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import Dict, List, Any, Type, Union, Optional, Tuple
-import os
-import json
-import copy
-from collections import defaultdict
-
-import pandas as pd
-
-
-from dash.dash import no_update
-from dash.dependencies import Input, Output, State
-from dash import dcc
-import dash_bootstrap_components as dbc
-from plotly.graph_objects import Figure
-from dash import html
-from dash.development.base_component import Component
-from ConfigSpace import ConfigurationSpace
-from dash.exceptions import PreventUpdate
-
-from deepcave import app
-from deepcave import c
-from deepcave.utils.data_structures import update_dict
-from deepcave.utils.logs import get_logger
-from deepcave.layouts.layout import Layout
-from deepcave.runs.handler import handler
-from deepcave.utils.layout import get_select_options
-
-
-logger = get_logger(__name__)
-
-
-class Plugin(Layout):
-    def __init__(self):
-        self.inputs = []
-        self.outputs = []
-
-        # Processing right now?
-        self.blocked = False
-
-        # Alert texts
-        self.alert_text = ""
-        self.alert_color = "success"
-        self.alert_update_required = False
-
-        super().__init__()
-
-    @staticmethod
-    @abstractmethod
-    def id():
-        raise NotImplementedError()
-
-    @staticmethod
-    def category():
-        return None
-
-    @staticmethod
-    def position():
-        return 99999
-
-    @staticmethod
-    @abstractmethod
-    def name():
-        raise NotImplementedError()
-
-    @staticmethod
-    def description() -> str:
-        return ''
-
-    @staticmethod
-    def button_caption():
-        return "Process"
-
-    @staticmethod
-    def check_requirements(runs, groups):
-        """
-        Returns either bool or str. If str, it is shown to the user.
-        """
-        return True
-
-    @staticmethod
-    def activate_run_selection() -> bool:
-        """
-        Shows a dropdown to select a run in the inputs layout. This feature is useful if only one run could be viewed at
-        a time. Moreover, it prevents the plugin to calculate results across all runs.
-
-        The run can be selected by inputs["run_name"]["value"].
-
-        Returns:
-            bool: True if run selection should be shown.
-        """
-
-        return False
-
-    def register_input(self, id, attributes=["value"], filter=False):
-        if isinstance(attributes, str):
-            attributes = [attributes]
-
-        for attribute in attributes:
-            key = (id, attribute, filter)
-            if key not in self.inputs:
-                self.inputs.append(key)
-
-        # We have to rearrange the inputs because `State`
-        # must follow all `Input`. Since all filters are `Input`, we have to
-        # shift them to the front.
-        self.inputs.sort(key=lambda x: x[2], reverse=True)
-
-        return self.get_internal_input_id(id)
-
-    def register_output(self, id, attribute="value", mpl=False):
-        assert isinstance(attribute, str)
-
-        if mpl:
-            id = id + "-mpl"
-
-        key = (id, attribute, mpl)
-        if key not in self.outputs:
-            self.outputs.append(key)
-
-        return self.get_internal_output_id(id)
-
-    def get_internal_id(self, id):
-        return self.id() + "-" + id
-
-    def get_internal_input_id(self, id):
-        return self.id() + "-" + id + "-input"
-
-    def get_internal_output_id(self, id):
-        return self.id() + "-" + id + "-output"
-
-    def register_callbacks(self):
-        # We have to call the output layout one time to register
-        # the values
-        # Problem: Inputs/Outputs can't be changed afterwards anymore.
-
-        if self.__class__.activate_run_selection():
-            self.__class__.get_run_input_layout(self.register_input)
-
-        self.__class__.get_input_layout(self.register_input)
-        self.__class__.get_filter_layout(
-            lambda a, b: self.register_input(a, b, filter=True))
-        self.__class__.get_output_layout(self.register_output)
-        self.__class__.get_mpl_output_layout(
-            lambda a, b: self.register_output(a, b, mpl=True))
-
-        # Handles the initial and the cashed input values
-        outputs = []
-        # [Input(self.get_internal_id("update-button"), 'n_clicks')]
-        inputs = []
-
-        # Define also inputs if they are declared as interactive
-        for id, attribute, _ in self.inputs:
-            inputs.append(Input(self.get_internal_input_id(id), attribute))
-
-        for id, attribute, _ in self.inputs:
-            outputs.append(Output(self.get_internal_input_id(id), attribute))
-
-        if len(outputs) > 0:
-            @app.callback(outputs, inputs)
-            def plugin_input_update(*inputs_list):
-                init = True
-                # Simple check if page was loaded for the first time
-                for input in inputs_list:
-                    if input is not None:
-                        init = False
-                        break
-
-                # Reload our inputs
-                if init:
-                    inputs = c.get("last_inputs", self.id())
-
-                    if inputs is None:
-                        inputs = self.__class__.load_inputs(self.runs)
-
-                        # Also update the run selection
-                        if self.__class__.activate_run_selection():
-                            new_inputs = self.__class__.load_run_inputs(
-                                self.runs)
-                            update_dict(inputs, new_inputs)
-
-                        # Set not used inputs
-                        for (id, attribute, _) in self.inputs:
-                            if id not in inputs:
-                                inputs[id] = {}
-
-                            if attribute not in inputs[id]:
-                                inputs[id][attribute] = None
-                else:
-                    # Map the list `inputs` to a dict.
-                    inputs = self._list_to_dict(inputs_list)
-
-                    if len(self.previous_inputs) == 0:
-                        self.previous_inputs = inputs.copy()
-
-                    # Only work on copies.
-                    # We don't want the inputs dict to be changed by the user.
-                    _previous_inputs = self.previous_inputs.copy()
-                    _inputs = inputs.copy()
-
-                    if self.__class__.activate_run_selection():
-                        if "run_name" in _previous_inputs:
-                            _previous_run_name = _previous_inputs["run_name"]["value"]
-                        else:
-                            _previous_run_name = None
-                        _run_name = inputs["run_name"]["value"]
-
-                        # Reset everything if run name changed.
-                        if _previous_run_name is not None and _previous_run_name != _run_name:
-                            # We can't use load_inputs here only
-                            # because `run_name` would be removed.
-                            # Also: We want to keep the current run name.
-                            update_dict(
-                                _inputs, self.__class__.load_inputs(self.runs))
-
-                            # TODO: Reset only inputs which are not available in another ru.
-                            # E.g. if options from budget in run_2 and run_3 are the same
-                            # take the budget from run_2 if changed to run_3. Otherwise, reset budgets.
-
-                    # How to update only parameters which have a dependency?
-                    user_dependencies_inputs = self.__class__.load_dependency_inputs(
-                        self.runs, _previous_inputs, _inputs)
-
-                    # Update dict
-                    # dict.update() remove keys, so it's done manually
-                    update_dict(inputs, user_dependencies_inputs)  # inplace
-
-                # From dict to list
-                inputs_list = self._dict_to_list(inputs, input=True)
-                self.previous_inputs = inputs
-
-                return inputs_list
-
-        # Update internal alert state to divs
-        @app.callback(
-            Output(self.get_internal_id("alert"), 'children'),
-            Output(self.get_internal_id("alert"), 'color'),
-            Output(self.get_internal_id("alert"), 'is_open'),
-            Input(self.get_internal_id("alert-interval"), 'n_intervals'))
-        def update_alert(_):
-            if self.alert_update_required:
-                self.alert_update_required = False
-                return self.alert_text, self.alert_color, True
-            else:
-                raise PreventUpdate()
-
-    def update_alert(self, text, color="success"):
-        self.alert_text = text
-        self.alert_color = color
-        self.alert_update_required = True
-
-    def _inputs_changed(self, inputs, last_inputs):
-        # Check if last_inputs are the same as the given inputs.
-        inputs_changed = False
-        filters_changed = False
-
-        # If only filters changed, then we don't need to
-        # calculate the results again.
-        if last_inputs is not None:
-            for (id, attribute, filter) in self.inputs:
-
-                if self.__class__.activate_run_selection():
-                    if id == "run_name":
-                        continue
-
-                if inputs[id][attribute] != last_inputs[id][attribute]:
-                    if not filter:
-                        inputs_changed = True
-                    else:
-                        filters_changed = True
-
-        return inputs_changed, filters_changed
-
-    def _process_raw_outputs(self, inputs, raw_outputs):
-        logger.debug("Process raw outputs.")
-
-        # Use raw outputs to update our layout
-        mpl_active = c.get("matplotlib-mode")
-        if mpl_active:
-            outputs = self.__class__.load_mpl_outputs(
-                inputs, raw_outputs, self.groups)
-        else:
-            outputs = self.__class__.load_outputs(
-                inputs, raw_outputs, self.groups)
-
-        if outputs == PreventUpdate:
-            raise PreventUpdate
-
-        # Map outputs here because it may be that the outputs are
-        # differently sorted than the values were registered.
-        if isinstance(outputs, dict):
-            outputs = self._dict_to_list(outputs, input=False)
-        else:
-            if not isinstance(outputs, list):
-                outputs = [outputs]
-
-        # We have to add no_updates here for the mode we don't want
-        count_outputs = 0
-        count_mpl_outputs = 0
-        for (_, _, mpl_mode) in self.outputs:
-            if mpl_mode:
-                count_mpl_outputs += 1
-            else:
-                count_outputs += 1
-
-        if mpl_active:
-            outputs = [no_update for _ in range(count_outputs)] + outputs
-        else:
-            outputs = outputs + [no_update for _ in range(count_mpl_outputs)]
-
-        return outputs
-
-    def _list_to_dict(self, values: list, input=True):
-        """
-        Maps the given values to a dict, regarding the sorting from
-        either self.inputs or self.outputs.
-
-        Returns:
-            dict
-        """
-
-        if input:
-            order = self.inputs
-        else:
-            order = self.outputs
-
-        mapping = {}
-        for value, (id, attribute, *_) in zip(values, order):
-            if id not in mapping:
-                mapping[id] = {}
-
-            mapping[id][attribute] = value
-
-        return mapping
-
-    def _dict_to_list(self, d, input=False):
-        """
-        Maps the given dict to a list, regarding the sorting from either
-        self.inputs or self.outputs.
-
-        Returns:
-            list.
-        """
-
-        if input:
-            order = self.inputs
-        else:
-            order = self.outputs
-
-        result = []
-        for (id, attribute, instance) in order:
-
-            if not input:
-                # Instance is mlp_mode in case of outputs
-                # Simply ignore other outputs.
-                if instance != c.get("matplotlib-mode"):
-                    continue
-            try:
-                value = d[id][attribute]
-                result += [value]
-            except:
-                result += [None]
-
-        return result
-
-    def _dict_as_key(self, d, remove_filters=False):
-        """
-        Converts a dictionary to a key. Only ids from self.inputs are considered.
-
-        Parameters:
-            d (dict): Dictionary to get the key from.
-            remove_filters (bool): Option wheather the filters should be included or not.
-
-        Returns:
-            key (str): Key as string from the given dictionary.
-        """
-
-        if not isinstance(d, dict):
-            return None
-
-        new_d = copy.deepcopy(d)
-        if remove_filters:
-            for (id, _, filter) in self.inputs:
-                if filter:
-                    if id in new_d:
-                        del new_d[id]
-
-        return str(new_d)
-
-    def __call__(self, render_button=False):
-        """
-        We overwrite the get_layout method here as we use a different
-        interface compared to layout.
-        """
-
-        self.previous_inputs = {}
-        self.runs = handler.get_runs()
-        groups = handler.get_groups()
-
-        if len(groups) == 0:
-            # Create groups with run_name: run_name
-            for run_name in self.runs.keys():
-                groups[run_name] = [run_name]
-
-        self.groups = groups
-
-        components = [html.H1(self.name())]
-        if self.description() != '':
-            components += [html.P(self.description())]
-
-        # Register alerts
-        components += [
-            dcc.Interval(
-                id=self.get_internal_id("alert-interval"),
-                interval=1*500,
-                n_intervals=5
-            ),
-            dbc.Alert(
-                id=self.get_internal_id("alert"),
-                is_open=False,
-                dismissable=True,
-                fade=True),
-        ]
-
-        status = self.check_requirements(self.runs, self.groups)
-        if isinstance(status, str):
-            self.update_alert(status, color="danger")
-            return components
-        elif isinstance(status, bool):
-            if not status:
-                return components
-
-        if self.__class__.activate_run_selection():
-            run_input_layout = [
-                self.__class__.get_run_input_layout(self.register_input)]
-        else:
-            run_input_layout = []
-
-        input_layout = self.__class__.get_input_layout(self.register_input)
-
-        separator_layout = []
-        if input_layout and run_input_layout:
-            separator_layout.append(html.Hr())
-
-        input_control_layout = html.Div(
-            style={} if render_button else {"display": "none"},
-            className="mt-3 clearfix",
-            children=[
-                dbc.Button(
-                    children=self.button_caption(),
-                    id=self.get_internal_id("update-button"),
-                ),
-                html.Span(
-                    html.Em(id=self.get_internal_id("processing-info")),
-                    className="ms-3 align-baseline",
-                )
-            ]
-        )
-
-        # We always have to render it because of the button.
-        # Button tells us if the page was just loaded.
-        components += [html.Div(
-            id=f'{self.id()}-input',
-            className="shadow-sm p-3 mb-3 bg-white rounded-lg",
-            children=run_input_layout +
-            separator_layout +
-            input_layout +
-            [input_control_layout],
-            style={} if render_button or input_layout or run_input_layout else {"display": "none"}
-        )]
-
-        def register(a, b): return self.register_input(a, b, filter=True)
-        filter_layout = self.__class__.get_filter_layout(register)
-        if len(filter_layout) > 0:
-            components += [html.Div(
-                id=f'{self.id()}-filter',
-                className="shadow-sm p-3 mb-3 bg-white rounded-lg",
-                children=filter_layout
-            )]
-
-        output_layout = self.__class__.get_output_layout(self.register_output)
-        if output_layout:
-            components += [html.Div(
-                id=f'{self.id()}-output',
-                className="shadow-sm p-3 bg-white rounded-lg loading-container",
-                children=output_layout,
-                style={} if not c.get(
-                    "matplotlib-mode") else {"display": "none"}
-            )]
-
-        def register(a, b): return self.register_input(a, b, mpl=True)
-        output_layout = self.__class__.get_mpl_output_layout(register)
-        if output_layout:
-            components += [html.Div(
-                id=f'{self.id()}-mpl-output',
-                className="shadow-sm p-3 bg-white rounded-lg loading-container",
-                children=output_layout,
-                style={} if c.get(
-                    "matplotlib-mode") else {"display": "none"}
-            )]
-
-        return components
-
-    @staticmethod
-    def get_run_input_layout(register):
-        return html.Div([
-            dbc.Select(
-                id=register("run_name", ["options", "value"]),
-                placeholder="Select run ..."
-            ),
-        ])
-
-    @staticmethod
-    def load_run_inputs(runs):
-        return {
-            "run_name": {
-                "options": get_select_options(runs.keys()),
-                "value": None
-            }
-        }
-
-    @staticmethod
-    def load_inputs(runs):
-        return {}
-
-    @staticmethod
-    def load_dependency_inputs(runs, previous_inputs, inputs):
-        return inputs
-
-    @staticmethod
-    def get_input_layout(register):
-        return []
-
-    @staticmethod
-    def get_filter_layout(register):
-        return []
-
-    @staticmethod
-    def get_output_layout(register):
-        return []
-
-    @staticmethod
-    def get_mpl_output_layout(register):
-        return []
-
-    @staticmethod
-    def load_outputs(inputs, outputs, groups):
-        """
-        Returns:
-            list or PreventUpdate: List of outputs (for `get_output_layout`) or PreventUpdate if
-            certain conditions are not met.
-        """
-
-        return {}
-
-    @staticmethod
-    def load_mpl_outputs(inputs, outputs, groups):
-        return {}
-
-    @staticmethod
-    @abstractmethod
-    def process(run, inputs):
-        pass
diff --git a/deepcave/plugins/static_plugin.py b/deepcave/plugins/static_plugin/__init__.py
similarity index 50%
rename from deepcave/plugins/static_plugin.py
rename to deepcave/plugins/static_plugin/__init__.py
index f753e176..35603c35 100644
--- a/deepcave/plugins/static_plugin.py
+++ b/deepcave/plugins/static_plugin/__init__.py
@@ -1,56 +1,61 @@
-from abc import ABC, abstractmethod
-from typing import Dict, List, Any, Type, Union, Optional, Tuple
+from abc import ABC
 
+from enum import Enum
+
+from dash import dcc
 from dash.dash import no_update
 from dash.dependencies import Input, Output, State
-from dash import dcc
-import dash_bootstrap_components as dbc
-from dash import html
-from dash.development.base_component import Component
 from dash.exceptions import PreventUpdate
 
-from deepcave import app, queue, c, rc
-from deepcave.runs.handler import handler
-from deepcave.utils.logs import get_logger
-from deepcave.plugins.plugin import Plugin
+from deepcave import app, c, queue, rc
+from deepcave.plugins import Plugin
 
 
-logger = get_logger(__name__)
+class PluginState(Enum):
+    UNSET = -1
+    READY = 0
+    NEEDS_PROCESSING = 1
+    PROCESSING = 2
 
 
-class StaticPlugin(Plugin):
+class StaticPlugin(Plugin, ABC):
     """
-    Calculation with queue. Made for time consuimg tasks.
+    Calculation with queue. Made for time-consuming tasks.
     """
 
     def __init__(self):
-        self._state = None
+        self._state: PluginState = PluginState.UNSET
         self._refresh_required = True
 
         super().__init__()
 
     def register_callbacks(self):
         super().register_callbacks()
+        self._callback_inputs_changed()
+        self._callback_loop_update_status_label()
+        self._callback_loop_trigger_main_loop()
 
+    def _callback_inputs_changed(self):
+        # Plugin specific outputs
         outputs = []
         for id, attribute, _ in self.outputs:
             outputs.append(Output(self.get_internal_output_id(id), attribute))
 
         inputs = [
-            Input(self.get_internal_id("update-button"), 'n_clicks'),
-            Input(self.get_internal_id("update-interval-output"), 'data')
+            Input(self.get_internal_id("update-button"), "n_clicks"),
+            Input(self.get_internal_id("update-interval-output"), "data"),
         ]
 
+        # Get other plugin specific inputs that might change
         for id, attribute, _ in self.inputs:
-            inputs.append(
-                Input(self.get_internal_input_id(id), attribute))
+            inputs.append(Input(self.get_internal_input_id(id), attribute))
 
         # Register updates from inputs
         @app.callback(outputs, inputs)
         def plugin_process(n_clicks, _, *inputs_list):
             """
             Parameters:
-                state (int): From button.
+                n_clicks (int): From button.
                 *inputs_list: Values from user.
             """
 
@@ -60,32 +65,9 @@ def plugin_process(n_clicks, _, *inputs_list):
             # it's easier to access them.
             inputs = self._list_to_dict(inputs_list, input=True)
             inputs_key = self._dict_as_key(inputs, remove_filters=True)
-            last_inputs = c.get("last_inputs", self.id())
-
-            runs = self.runs
-
-            # Special case: If run selection is active
-            # Don't update anything if the inputs haven't changed
-            if self.__class__.activate_run_selection():
-                if inputs["run_name"]["value"] is None:
-                    self._blocked = False
-                    raise PreventUpdate
-
-                # Update runs
-                runs = {}
-                run_name = inputs["run_name"]["value"]
-                runs[run_name] = self.runs[run_name]
-                run_names = [run_name]
-
-                # Also:
-                # Remove `run_name` from inputs_key because
-                # we don't want the run names included.
-                _inputs = inputs.copy()
-                del _inputs["run_name"]
-
-                inputs_key = self._dict_as_key(_inputs, remove_filters=True)
-            else:
-                run_names = runs.keys()
+            last_inputs = c.get("last_inputs", self.id)
+
+            runs = self.get_selected_runs(inputs)
 
             button_pressed = n_clicks is not None
             inputs_changed = inputs != last_inputs
@@ -93,83 +75,90 @@ def plugin_process(n_clicks, _, *inputs_list):
             # Check current state
             raw_outputs = {}
             raw_outputs_available = True
-            for run_name in run_names:
-                raw_outputs[run_name] = rc[run_name].get(self.id(), inputs_key)
+            for run in runs:
+                raw_outputs[run.name] = rc.get_run(run).get(self.id, inputs_key)
 
-                if raw_outputs[run_name] is None:
+                if raw_outputs[run.name] is None:
                     raw_outputs_available = False
 
+            # Process
             if raw_outputs_available:
-                self._state = 0
+                # Load raw outputs from cache
+                self._state = PluginState.READY
 
                 if inputs_changed or self._refresh_required:
-                    c.set("last_inputs", self.id(), value=inputs)
+                    c.set("last_inputs", self.id, value=inputs)
 
                     outputs = self._process_raw_outputs(inputs, raw_outputs)
                     self._refresh_required = False
 
                     return outputs
             else:
+                # Load from process
                 self._state = 1
 
-                if button_pressed and self._state != 2:
-                    logger.debug("Button pressed.")
+                if button_pressed and self._state != PluginState.PROCESSING:
+                    self.logger.debug("Button pressed.")
 
                     # Check if we need to process
-                    for run_name, run in runs.items():
-                        job_id = self._get_job_id(run_name, inputs_key)
+                    for run in runs:
+                        job_id = self._get_job_id(run.name, inputs_key)
 
                         # We already got our results or it was already processed
-                        if raw_outputs[run_name] is not None or queue.is_processed(job_id):
+                        if raw_outputs[run.name] is not None or queue.is_processed(
+                            job_id
+                        ):
                             continue
 
-                        meta = {
-                            "display_name": self.name(),
-                            "run_name": run_name,
+                        job_meta = {
+                            "display_name": self.name,
+                            "run_name": run.name,
+                            "run_cache_id": run.run_cache_id,
                             "inputs_key": inputs_key,
                         }
 
-                        logger.debug(f"Enqueued {run_name}.")
+                        self.logger.debug(f"Enqueued {run.name}.")
 
                         # Start the task in rq
                         queue.enqueue(
-                            self.process,
-                            args=[run, inputs],
+                            self._process,
+                            args=[self.process, run.run_cache_id, inputs],
                             job_id=job_id,
-                            meta=meta
+                            meta=job_meta,
                         )
 
                     # Reset button
                     self._reset_button = True
-                    self._state = 2
+                    self._state = PluginState.PROCESSING
                 else:
                     # Get finished jobs and save them
                     for job in queue.get_finished_jobs():
+                        job_id = job.id
                         try:
-                            job_id = job.id
                             job_run_outputs = job.result
                             job_meta = job.meta
                             job_inputs_key = job_meta["inputs_key"]
-                            job_run_name = job_meta["run_name"]
+                            job_run_name = job_meta["run_cache_id"]
 
-                            logger.debug(f"Job `{job_id}`")
+                            self.logger.debug(f"Job `{job_id}`")
 
                             # Save results in cache
-                            rc[job_run_name].set(
-                                self.id(), job_inputs_key, value=job_run_outputs)
-                            logger.debug(f"... cached")
+                            rc.get(job_run_name).set(
+                                self.id, job_inputs_key, value=job_run_outputs
+                            )
+                            self.logger.debug(f"... cached")
 
                             queue.delete_job(job_id)
-                            logger.debug(f"... deleted")
+                            self.logger.debug(f"... deleted")
                         except:
                             queue.delete_job(job_id)
-                            logger.debug(f"... deleted")
+                            self.logger.debug(f"... deleted")
 
                     # Check if queue is still running
                     queue_running = False
                     queue_pending = False
-                    for run_name in run_names:
-                        job_id = self._get_job_id(run_name, inputs_key)
+                    for run in runs:
+                        job_id = self._get_job_id(run.name, inputs_key)
                         if queue.is_running(job_id):
                             queue_running = True
 
@@ -177,15 +166,16 @@ def plugin_process(n_clicks, _, *inputs_list):
                             queue_pending = True
 
                     if queue_running or queue_pending:
-                        self._state = 2
+                        self._state = PluginState.PROCESSING
 
             self._blocked = False
             raise PreventUpdate
 
-        output = Output(self.get_internal_id('update-interval-output'), 'data')
+    def _callback_loop_trigger_main_loop(self):
+        output = Output(self.get_internal_id("update-interval-output"), "data")
         inputs = [
-            Input(self.get_internal_id("update-interval"), 'n_intervals'),
-            State(self.get_internal_id("update-interval-output"), 'data'),
+            Input(self.get_internal_id("update-interval"), "n_intervals"),
+            State(self.get_internal_id("update-interval-output"), "data"),
         ]
 
         # Interval should not always run the main callback the whole time
@@ -197,35 +187,36 @@ def plugin_check_blocked(_, data):
                 raise PreventUpdate
 
             # This will trigger the main loop
-            return data+1
+            return data + 1
 
+    def _callback_loop_update_status_label(self):
         output = [
-            Output(self.get_internal_id("processing-info"), 'children'),
-            Output(self.get_internal_id("update-button"), 'n_clicks'),
+            Output(self.get_internal_id("processing-info"), "children"),
+            Output(self.get_internal_id("update-button"), "n_clicks"),
         ]
-        input = Input(self.get_internal_id("update-interval"), 'n_intervals')
+        input = Input(self.get_internal_id("update-interval"), "n_intervals")
 
         # Update status label
         # Register updates from inputs
         @app.callback(output, input)
         def plugin_update_status(_):
-            status = ""
-            if self._state == 0:
-                status = "Ready."
-            if self._state == 1:
-                status = "Processing Necessary."
-            if self._state == 2:
-                status = "Processing ..."
+            status_str = ""
+            if self._state == PluginState.READY:
+                status_str = "Ready."
+            elif self._state == PluginState.NEEDS_PROCESSING:
+                status_str = "Processing Necessary."
+            elif self._state == PluginState.PROCESSING:
+                status_str = "Processing ..."
 
             button = no_update
             if self._reset_button:
                 self._reset_button = False
                 button = None
 
-            return status, button
+            return status_str, button
 
-    def _get_job_id(self, run_name, inputs_key):
-        return run_name + "-" + inputs_key
+    def _get_job_id(self, run_name, inputs_key) -> str:
+        return f"{run_name}-{inputs_key}"
 
     def __call__(self):
         self._state = 1
@@ -234,10 +225,8 @@ def __call__(self):
         self._blocked = False
 
         components = [
-            dcc.Interval(
-                id=self.get_internal_id("update-interval"), interval=200),
-            dcc.Store(id=self.get_internal_id(
-                "update-interval-output"), data=0),
+            dcc.Interval(id=self.get_internal_id("update-interval"), interval=200),
+            dcc.Store(id=self.get_internal_id("update-interval-output"), data=0),
         ]
         components += super().__call__(True)
 
diff --git a/deepcave/plugins/fanova.py b/deepcave/plugins/static_plugin/fanova.py
similarity index 54%
rename from deepcave/plugins/fanova.py
rename to deepcave/plugins/static_plugin/fanova.py
index cb99b773..49b52a3b 100644
--- a/deepcave/plugins/fanova.py
+++ b/deepcave/plugins/static_plugin/fanova.py
@@ -1,91 +1,61 @@
-from typing import Dict, Type, Any
+from ast import literal_eval
 
-from dash import dcc, html
-from dash.exceptions import PreventUpdate
 import dash_bootstrap_components as dbc
-import plotly.graph_objs as go
-
-import pandas as pd
 import numpy as np
+import plotly.graph_objs as go
+from dash import dcc, html
+from dash.exceptions import PreventUpdate
 
+from deepcave import run_handler
+from deepcave.evaluators.fanova import fANOVA as _fANOVA
 from deepcave.plugins.static_plugin import StaticPlugin
-from deepcave.utils.logs import get_logger
+from deepcave.runs import AbstractRun
 from deepcave.utils.data_structures import update_dict
-from deepcave.utils.styled_plotty import get_color
-from deepcave.utils.layout import get_slider_marks, get_select_options, get_checklist_options
-from deepcave.utils.compression import serialize, deserialize
-
-from deepcave.evaluators.fanova import fANOVA as _fANOVA
-
-
-logger = get_logger(__name__)
+from deepcave.utils.layout import get_checklist_options
 
 
 class fANOVA(StaticPlugin):
-    @staticmethod
-    def id():
-        return "fanova"
+    id = "fanova"
+    name = "fANOVA"
+    icon = "far fa-star"
 
-    @staticmethod
-    def name():
-        return "fANOVA"
-
-    @staticmethod
-    def position():
-        return 100
-
-    @staticmethod
-    def category():
-        return "Hyperparameter Analysis"
-
-    @staticmethod
-    def activate_run_selection():
-        return True
+    activate_run_selection = True
 
     @staticmethod
     def get_input_layout(register):
         return [
             dbc.Label("Number of trees"),
-            dbc.Input(id=register("num_trees", "value"))
+            dbc.Input(id=register("num_trees", "value")),
         ]
 
     @staticmethod
     def get_filter_layout(register):
         return [
-            html.Div([
-                dbc.Label("Hyperparameters"),
-                dbc.Checklist(
-                    id=register("hyperparameters", ["options", "value"])),
-            ], className="mb-3"),
-
-            html.Div([
-                dbc.Label("Budgets"),
-                dbc.Checklist(
-                    id=register("budgets", ["options", "value"])),
-            ]),
+            html.Div(
+                [
+                    dbc.Label("Hyperparameters"),
+                    dbc.Checklist(id=register("hyperparameters", ["options", "value"])),
+                ],
+                className="mb-3",
+            ),
+            html.Div(
+                [
+                    dbc.Label("Budgets"),
+                    dbc.Checklist(id=register("budgets", ["options", "value"])),
+                ]
+            ),
         ]
 
-    @staticmethod
-    def load_inputs(runs):
+    def load_inputs(self):
         return {
-            "num_trees": {
-                "value": 16
-            },
-            "hyperparameters": {
-                "options": get_checklist_options(),
-                "value": []
-            },
-            "budgets": {
-                "options": get_checklist_options(),
-                "value": []
-            },
+            "num_trees": {"value": 16},
+            "hyperparameters": {"options": get_checklist_options(), "value": []},
+            "budgets": {"options": get_checklist_options(), "value": []},
         }
 
-    @staticmethod
-    def load_dependency_inputs(runs, previous_inputs, inputs):
-        run = runs[inputs["run_name"]["value"]]
-        budgets = run.get_budgets()
-        hp_names = run.configspace.get_hyperparameter_names()
+    def load_dependency_inputs(self, previous_inputs, inputs, selected_run=None):
+        budgets = selected_run.get_budgets(human=True)
+        hp_names = selected_run.configspace.get_hyperparameter_names()
 
         new_inputs = {
             "hyperparameters": {
@@ -99,6 +69,8 @@ def load_dependency_inputs(runs, previous_inputs, inputs):
 
         # Restrict to three hyperparameters
         num_trees = inputs["num_trees"]["value"]
+
+        # Reset invalid values
         try:
             int(num_trees)
         except:
@@ -107,7 +79,7 @@ def load_dependency_inputs(runs, previous_inputs, inputs):
         return inputs
 
     @staticmethod
-    def process(run, inputs):
+    def process(run: AbstractRun, inputs):
         hp_names = run.configspace.get_hyperparameter_names()
         budgets = run.get_budgets()
 
@@ -117,14 +89,14 @@ def process(run, inputs):
             X, Y = run.get_encoded_configs(budget=budget, for_tree=True)
 
             evaluator = _fANOVA(
-                X, Y,
+                X,
+                Y,
                 configspace=run.configspace,
-                num_trees=int(inputs["num_trees"]["value"])
+                num_trees=int(inputs["num_trees"]["value"]),
             )
             importance_dict = evaluator.quantify_importance(
-                hp_names,
-                depth=1,
-                sorted=False)
+                hp_names, depth=1, sort=False
+            )
 
             importance_dict = {k[0]: v for k, v in importance_dict.items()}
 
@@ -134,27 +106,27 @@ def process(run, inputs):
 
     @staticmethod
     def get_output_layout(register):
-        return [
-            dcc.Graph(register("graph", "figure"))
-        ]
+        return [dcc.Graph(register("graph", "figure"))]
 
     @staticmethod
     def load_outputs(inputs, outputs, _):
-        run_name = inputs["run_name"]["value"]
-        outputs = outputs[run_name]
+        run = run_handler.from_run_id(inputs["run_name"]["value"])
+        outputs = outputs[run.name]
+
         # First selected, should always be shown first
         selected_hyperparameters = inputs["hyperparameters"]["value"]
         selected_budgets = inputs["budgets"]["value"]
 
         if len(selected_hyperparameters) == 0 or len(selected_budgets) == 0:
-            return PreventUpdate
+            raise PreventUpdate()
 
         # TODO: After json serialize/deserialize, budget is not an integer anymore
-        convert_type = type(selected_budgets[0])
+        convert_type = lambda x: str(float(x))  # type(selected_budgets[0])
 
         # Collect data
         data = {}
         for budget, importance_dict in outputs.items():
+            budget = literal_eval(budget) if isinstance(budget, str) else budget
             budget = convert_type(budget)
 
             if budget not in inputs["budgets"]["value"]:
@@ -171,30 +143,27 @@ def load_outputs(inputs, outputs, _):
                 y += [results[1]]
                 error_y += [results[3]]
 
-            data[budget] = (
-                np.array(x),
-                np.array(y),
-                np.array(error_y)
-            )
+            data[budget] = (np.array(x), np.array(y), np.array(error_y))
 
             # if filters["sort"]["value"] == fidelity_id:
             #    selected_fidelity = fidelity
 
         # Sort by last fidelity now
         last_selected_budget = selected_budgets[-1]
-        idx = np.argsort(
-            data[last_selected_budget][1], axis=None)[::-1]
+        idx = np.argsort(data[last_selected_budget][1], axis=None)[::-1]
 
         bar_data = []
         for budget, values in data.items():
-            bar_data += [go.Bar(
-                name=budget,
-                x=values[0][idx],
-                y=values[1][idx],
-                error_y_array=values[2][idx])
+            bar_data += [
+                go.Bar(
+                    name=budget,
+                    x=values[0][idx],
+                    y=values[1][idx],
+                    error_y_array=values[2][idx],
+                )
             ]
 
         fig = go.Figure(data=bar_data)
-        fig.update_layout(barmode='group')
+        fig.update_layout(barmode="group")
 
         return [fig]
diff --git a/deepcave/queue.py b/deepcave/queue.py
index 84d51f14..40602f7b 100644
--- a/deepcave/queue.py
+++ b/deepcave/queue.py
@@ -1,3 +1,5 @@
+from typing import Any, Callable
+
 import redis
 from rq import Queue as _Queue
 from rq import Worker
@@ -9,9 +11,9 @@
 
 
 class Queue:
-    def __init__(self, redis_url="redis://localhost:6379"):
-        self._connection = redis.from_url(redis_url)
-        self._queue = _Queue('high', connection=self._connection)
+    def __init__(self, address: str, port: int):
+        self._connection = redis.from_url(address + ":" + str(port))
+        self._queue = _Queue("high", connection=self._connection)
 
     def ready(self):
         # Check if at least one worker is in use:
@@ -23,7 +25,11 @@ def ready(self):
         return False
 
     def is_processed(self, job_id):
-        if self.is_running(job_id) or self.is_pending(job_id) or self.is_finished(job_id):
+        if (
+            self.is_running(job_id)
+            or self.is_pending(job_id)
+            or self.is_finished(job_id)
+        ):
             return True
 
         return False
@@ -49,7 +55,7 @@ def is_finished(self, job_id):
 
         return False
 
-    def get_jobs(self, registry="running"):
+    def get_jobs(self, registry="running") -> list[Job]:
         if registry == "running":
             registry = self._queue.started_job_registry
         elif registry == "pending":
@@ -66,20 +72,20 @@ def get_jobs(self, registry="running"):
 
         return results
 
-    def get_running_jobs(self):
+    def get_running_jobs(self) -> list[Job]:
         return self.get_jobs(registry="running")
 
-    def get_pending_jobs(self):
+    def get_pending_jobs(self) -> list[Job]:
         return self.get_jobs(registry="pending")
 
-    def get_finished_jobs(self):
+    def get_finished_jobs(self) -> list[Job]:
         return self.get_jobs(registry="finished")
 
-    def delete_job(self, job_id):
+    def delete_job(self, job_id: str):
         registries = [
             self._queue.finished_job_registry,
             self._queue,
-            self._queue.started_job_registry
+            self._queue.started_job_registry,
         ]
 
         for r in registries:
@@ -88,7 +94,9 @@ def delete_job(self, job_id):
             except:
                 pass
 
-    def enqueue(self, func, args, job_id, meta):
+    def enqueue(
+        self, func: Callable[[Any], Any], args: Any, job_id: str, meta: dict[str, str]
+    ):
         # First check if job_id is already in use
         if self.is_processed(job_id):
             logger.debug("Job was not added because it was processed already.")
@@ -99,7 +107,7 @@ def enqueue(self, func, args, job_id, meta):
             args=args,
             job_id=job_id,
             meta=meta,
-            result_ttl=-1  # Make sure it's not automatically deleted.
+            result_ttl=-1,  # Make sure it's not automatically deleted.
         )
 
     def __getattr__(self, name):
diff --git a/deepcave/runs/__init__.py b/deepcave/runs/__init__.py
index e69de29b..7c6d7162 100644
--- a/deepcave/runs/__init__.py
+++ b/deepcave/runs/__init__.py
@@ -0,0 +1,544 @@
+from abc import ABC, abstractmethod
+from typing import Any, Dict, Iterable, Iterator, List, Optional, Tuple, Union
+
+import copy
+from dataclasses import dataclass
+from enum import IntEnum
+
+import ConfigSpace
+import numpy as np
+import pandas as pd
+from ConfigSpace import (
+    CategoricalHyperparameter,
+    Configuration,
+    Constant,
+    UniformFloatHyperparameter,
+    UniformIntegerHyperparameter,
+)
+
+from deepcave.runs.objective import Objective
+from deepcave.utils.hash import string_to_hash
+from deepcave.utils.logs import get_logger
+
+
+class NotValidRunError(Exception):
+    """Raised if directory is not a valid run."""
+
+    pass
+
+
+class NotMergeableError(Exception):
+    """Raised if two or more runs are not mergeable"""
+
+    pass
+
+
+class Status(IntEnum):
+    SUCCESS = 1
+    TIMEOUT = 2
+    MEMORYOUT = 3
+    CRASHED = 4
+    ABORTED = 5
+    RUNNING = 6
+
+
+@dataclass
+class Trial:
+    config_id: str
+    budget: int
+    costs: float
+    start_time: float
+    end_time: float
+    status: Status
+    additional: dict[str, Any]
+
+    def __post_init__(self):
+        if isinstance(self.status, int):
+            self.status = Status(self.status)
+
+        assert isinstance(self.status, Status)
+
+    def get_key(self) -> Tuple[str, int]:
+        return self.config_id, self.budget  # noqa
+
+
+class AbstractRun(ABC):
+    prefix: str
+
+    def __init__(self, name: str) -> None:
+        self.name = name
+        self.logger = get_logger(self.__class__.__name__)
+
+        # objects created by reset
+        self.reset()
+
+    def reset(self) -> None:
+        self.meta: Dict[str, Any] = {}
+        self.configspace: Optional[ConfigSpace.ConfigurationSpace] = None
+        self.configs: Dict[int, Configuration] = {}
+        self.origins: Dict[int, str] = {}
+        self.models: Dict[int, Any] = {}
+
+        self.history: List[Trial] = []
+        self.trial_keys: Dict[Tuple[str, int], int] = {}
+
+    @property
+    def run_cache_id(self) -> str:
+        return string_to_hash(f"{self.prefix}:{self.name}")
+
+    @property
+    @abstractmethod
+    def hash(self) -> str:
+        """
+        Hash of current run. If hash changes, cache has to be cleared, as something has changed
+        """
+        pass
+
+    def get_meta(self) -> None:
+        return self.meta
+
+    def empty(self) -> None:
+        return len(self.history) == 0
+
+    def get_objectives(self) -> None:
+        objectives = []
+        for d in self.meta["objectives"]:
+            objective = Objective(
+                name=d["name"],
+                lower=d["lower"],
+                upper=d["upper"],
+                optimize=d["optimize"],
+            )
+
+            objective["lock_lower"] = d["lock_lower"]
+            objective["lock_upper"] = d["lock_upper"]
+
+            objectives.append(objective)
+
+        return objectives
+
+    def get_trials(self) -> Iterator[Trial]:
+        yield from self.history
+
+    def get_objective_name(self, objective_names=None) -> str:
+        """
+        Get the cost name of given objective names.
+        Returns "Combined Cost" if multiple objective names were involved.
+        """
+
+        given_objective_names = self.get_objective_names()
+
+        if objective_names is None:
+            if len(given_objective_names) == 1:
+                return given_objective_names[0]
+        else:
+            if isinstance(objective_names, str):
+                objective_names = [objective_names]
+
+            if len(objective_names) == 1:
+                return objective_names[0]
+
+        return "Combined Cost"
+
+    def get_objective_names(self) -> List:
+        return [obj["name"] for obj in self.get_objectives()]
+
+    def get_configs(self, budget=None) -> List:
+        configs = []
+        for trial in self.history:
+            if budget is not None:
+                if budget != trial.budget:
+                    continue
+
+            config = self.configs[trial.config_id]
+            configs += [config]
+
+        return configs
+
+    def get_config(self, id):
+        return self.configs[id]
+
+    def get_config_id(self, config: dict):
+        # Find out config id
+        for id, c in self.configs.items():
+            if c == config:
+                return id
+
+        return None
+
+    def get_budget(self, id: int) -> float:
+        return self.meta["budgets"][id]
+
+    def get_budgets(self, human=False) -> List[str]:
+        """
+        There's at least one budget with None included.
+
+        Args:
+
+        """
+        budgets = self.meta["budgets"]
+        assert len(budgets) > 0
+
+        if human:
+            readable_budgets = []
+            for b in budgets:
+                if b is None:
+                    readable_budgets += ["None"]
+                else:
+                    readable_budgets += [str(np.round(float(b), 2))]
+
+            return readable_budgets
+
+        return budgets
+
+    def get_highest_budget(self):
+        budgets = self.meta["budgets"]
+        if len(budgets) == 0:
+            return None
+
+        return budgets[-1]
+
+    def _process_costs(self, costs: Iterable[float]) -> list[float]:
+        """
+        Get rid of none costs.
+        """
+
+        new_costs = []
+        for cost, obj in zip(costs, self.get_objectives()):
+            # Replace with the worst cost
+            if cost is None:
+                if obj["optimize"] == "lower":
+                    cost = obj["upper"]
+                else:
+                    cost = obj["lower"]
+
+            new_costs += [cost]
+
+        return new_costs
+
+    def get_costs(self, budget=None, statuses=None):
+        """
+        If no budget is given, the highest budget is chosen.
+        """
+
+        if budget is None:
+            budget = self.get_highest_budget()
+
+        results = {}
+        for trial in self.history:
+            if trial.budget is not None:
+                if trial.budget != budget:
+                    continue
+
+            if statuses is not None:
+                if trial.status not in statuses:
+                    continue
+
+            results[trial.config_id] = self._process_costs(trial.costs)
+
+        return results
+
+    def get_min_cost(self, objective_names=None, budget=None, statuses=None):
+        min_cost = np.inf
+        best_config = None
+
+        results = self.get_costs(budget, statuses)
+        for config_id, costs in results.items():
+            cost = self.calculate_cost(costs, objective_names, normalize=True)
+
+            if cost < min_cost:
+                min_cost = cost
+                best_config = self.get_config(config_id)
+
+        return min_cost, best_config
+
+    def calculate_cost(self, costs, objective_names=None, normalize=False) -> float:
+        """
+        Calculates cost from multiple costs.
+        Normalizes the cost first and weight every cost the same.
+        """
+
+        costs = self._process_costs(costs)
+
+        if objective_names is None:
+            objective_names = self.get_objective_names()
+
+        assert len(objective_names) > 0
+
+        # No normalization needed
+        if len(objective_names) == 1 and not normalize:
+            return costs[0]
+
+        objectives = self.get_objectives()
+
+        # First normalize
+        filtered_objectives = []
+        normalized_costs = []
+        for cost, objective in zip(costs, objectives):
+            if objective["name"] not in objective_names:
+                continue
+
+            a = cost - objective["lower"]
+            b = objective["upper"] - objective["lower"]
+            normalized_cost = a / b
+
+            # We optimize the lower
+            # So we need to flip the normalized cost
+            if objective["optimize"] == "upper":
+                normalized_cost = 1 - normalized_cost
+
+            normalized_costs.append(normalized_cost)
+            filtered_objectives.append(objective)
+
+        # Give the same weight to all objectives (for now)
+        objective_weights = [
+            1 / len(objectives) for _ in range(len(filtered_objectives))
+        ]
+
+        costs = [u * v for u, v in zip(normalized_costs, objective_weights)]
+        cost = np.mean(costs).item()
+
+        return cost
+
+    def get_trajectory(self, objective_names=None, budget=None):
+        if budget is None:
+            budget = self.get_highest_budget()
+
+        costs_mean = []
+        costs_std = []
+        ids = []
+        times = []
+
+        order = []
+        # Sort self.history by end_time
+        for id, trial in enumerate(self.history):
+            order.append((id, trial.end_time))
+
+        order.sort(key=lambda tup: tup[1])
+
+        current_cost = np.inf
+        for id, cost in order:
+            trial = self.history[id]
+            # Only consider selected/last budget
+            if trial.budget != budget:
+                continue
+
+            cost = self.calculate_cost(trial.costs, objective_names)
+            if cost < current_cost:
+                current_cost = cost
+
+                costs_mean.append(cost)
+                costs_std.append(0)
+                times.append(trial.end_time)
+                ids.append(id)
+
+        return times, costs_mean, costs_std, ids
+
+    def get_encoded_configs(
+        self,
+        objective_names=None,
+        budget=None,
+        statuses=None,
+        for_tree=False,
+        pandas=False,
+    ):
+        """
+        Args:
+            for_tree (bool): Inactives are treated differently. If false, all inactives are set to
+            -1.
+            normalize (bool): Normalize the configuration between 0 and 1.
+            pandas (bool): Return pandas DataFrame instead of X and Y.
+        Returns:
+            X, Y (np.array): Encoded configurations OR
+            df, df_labels (pd.DataFrame): Encoded dataframes if pandas equals True.
+        """
+
+        hp_names = self.configspace.get_hyperparameter_names()
+
+        X, Y = [], []
+        labels = []
+
+        results = self.get_costs(budget, statuses)
+        for config_id, costs in results.items():
+            config = self.configs[config_id]
+            config = Configuration(self.configspace, config)
+
+            y = self.calculate_cost(costs, objective_names)
+            x = config.get_array()
+
+            X.append(x)
+            Y.append(y)
+
+            labels_ = []
+            for hp_name in hp_names:
+                # hyperparameter name may not be in config
+                if hp_name in config:
+                    label = config[hp_name]
+
+                    # Scientific notation
+                    if type(label) == float:
+                        if str(label).startswith("0.000") or "e-" in str(label):
+                            label = np.format_float_scientific(label, precision=2)
+                        else:
+                            # Round to 2 decimals
+                            label = np.round(label, 2)
+
+                    labels_ += [label]
+                else:
+                    labels_ += ["NaN"]
+
+            # We append y here directly
+            labels_ += [y]
+            labels.append(labels_)
+
+        X = np.array(X)
+        Y = np.array(Y)
+
+        # Imputation: Easiest case is to replace all nans with -1
+        # However, since Stefan used different values for inactives
+        # we also have to use different inactives to be compatible
+        # with the random forests.
+        # https://github.com/automl/SMAC3/blob/a0c89502f240c1205f83983c8f7c904902ba416d/smac/epm/base_rf.py#L45
+
+        if not for_tree:
+            X[np.isnan(X)] = -1
+        else:
+            conditional = {}
+            impute_values = {}
+
+            for idx, hp in enumerate(self.configspace.get_hyperparameters()):
+                if idx not in conditional:
+                    parents = self.configspace.get_parents_of(hp.name)
+                    if len(parents) == 0:
+                        conditional[idx] = False
+                    else:
+                        conditional[idx] = True
+                        if isinstance(hp, CategoricalHyperparameter):
+                            impute_values[idx] = len(hp.choices)
+                        elif isinstance(
+                            hp,
+                            (UniformFloatHyperparameter, UniformIntegerHyperparameter),
+                        ):
+                            impute_values[idx] = -1
+                        elif isinstance(hp, Constant):
+                            impute_values[idx] = 1
+                        else:
+                            raise ValueError
+
+                if conditional[idx] is True:
+                    nonfinite_mask = ~np.isfinite(X[:, idx])
+                    X[nonfinite_mask, idx] = impute_values[idx]
+
+        if pandas:
+            cost_column = self.get_objective_name(objective_names)
+            columns = [name for name in self.configspace.get_hyperparameter_names()] + [
+                cost_column
+            ]
+
+            Y = Y.reshape(-1, 1)
+            data = np.concatenate((X, Y), axis=1)
+
+            df = pd.DataFrame(data=data, columns=columns)
+            df_labels = pd.DataFrame(data=labels, columns=columns)
+
+            return df, df_labels
+
+        return X, Y
+
+
+def check_equality(
+    runs: List[AbstractRun],
+    meta: bool = True,
+    configspace: bool = True,
+    objectives: bool = True,
+    budgets: bool = True,
+) -> Dict[str, Any]:
+    """
+    Checks the passed runs on equality based on the selected runs and returns the requested
+    attributes.
+
+    Parameters
+    ----------
+    runs : list[AbstractRun]
+        Runs to check for equality.
+    meta : bool, optional
+        Meta-Data excluding objectives and budgets, by default True
+    configspace : bool, optional
+        ConfigSpace, by default True
+    objectives : bool, optional
+        Objectives, by default True
+    budgets : bool, optional
+        Budgets, by default True
+
+    Returns
+    -------
+    Dict[str, Any]
+        Dictionary containing the checked attributes.
+    """
+
+    result = {}
+
+    if len(runs) == 0:
+        return result
+
+    # Check meta
+    if meta:
+        m1 = runs[0].get_meta()
+        for run in runs:
+            m2 = run.get_meta()
+
+            for k, v in m1.items():
+                # Don't check on objectives or budgets
+                if k == "objectives" or k == "budgets":
+                    continue
+
+                if k not in m2 or m2[k] != v:
+                    raise NotMergeableError("Meta data of runs are not equal.")
+
+        result["meta"] = m1
+
+    # Make sure the same configspace is used
+    # Otherwise it does not make sense to merge
+    # the histories
+    if configspace:
+        cs1 = runs[0].configspace
+        for run in runs:
+            cs2 = run.configspace
+            if cs1 != cs2:
+                raise NotMergeableError("Configspace of runs are not equal.")
+
+        result["configspace"] = cs1
+
+    # Also check if budgets are the same
+    if budgets:
+        b1 = runs[0].get_budgets()
+        for run in runs:
+            b2 = run.get_budgets()
+            if b1 != b2:
+                raise NotMergeableError("Budgets of runs are not equal.")
+
+        result["budgets"] = b1
+        if meta:
+            result["meta"]["budgets"] = b1
+
+    # And if objectives are the same
+    if objectives:
+        o1 = None
+        for run in runs:
+            o2 = run.get_objectives()
+
+            if o1 is None:
+                o1 = o2
+                continue
+
+            if len(o1) != len(o2):
+                raise NotMergeableError("Objectives of runs are not equal.")
+
+            for o1_, o2_ in zip(o1, o2):
+                o1_.merge(o2_)
+
+        result["objectives"] = o1
+        if meta:
+            result["meta"]["objectives"] = o1
+
+    return result
diff --git a/deepcave/runs/converters/bohb.py b/deepcave/runs/converters/bohb.py
index 945ac85f..6925a7f9 100644
--- a/deepcave/runs/converters/bohb.py
+++ b/deepcave/runs/converters/bohb.py
@@ -1,56 +1,48 @@
-import os
-import json
-import glob
-import pandas as pd
-from typing import Dict, Type, Any
-
-import ConfigSpace
-from deepcave.runs.run import Status
-from deepcave.runs.converters.converter import Converter
-from deepcave.runs.run import Run
+from pathlib import Path
+
+from deepcave.runs import Status
 from deepcave.runs.objective import Objective
+from deepcave.runs.run import Run
 from deepcave.utils.hash import file_to_hash
 
 
-class BOHB(Converter):
-    @staticmethod
-    def name() -> str:
-        return "BOHB"
+class BOHBRun(Run):
+    prefix = "BOHB"
+    _initial_order = 2
 
-    def get_run_id(self, working_dir, run_name) -> str:
+    @property
+    def hash(self) -> str:
         """
-        The id from the files in the current working_dir/run_name/*. For example, history.json could be read and hashed.
-        Idea behind: If id changed, then we have to update cached trials.
+        The id from the files in the current working_dir/run_name/*. For example, results.json could be read and hashed.
+        Idea behind: If id changed, then we have to update cached run.
         """
+        # Use hash of results.json as id
+        return file_to_hash(self.path / "results.json")
 
-        # Use hash of history.json as id
-        return file_to_hash(os.path.join(working_dir, run_name, "results.json"))
-
-    def get_run(self, working_dir, run_name) -> Run:
+    @classmethod
+    def from_path(cls, path: Path) -> "BOHBRun":
         """
-        Based on working_dir/run_name/*, return a new trials object.
+        Based on path, return a new run object.
         """
 
-        base = os.path.join(working_dir, run_name)
-
         # Read configspace
         from ConfigSpace.read_and_write import json as cs_json
-        with open(os.path.join(base, 'configspace.json'), 'r') as f:
-            configspace = cs_json.read(f.read())
+
+        configspace = cs_json.read((path / "configspace.json").read_text())
 
         # Read objectives
         # We have to define it ourselves, because we don't know the type of the objective
         # Only lock lower
         objective = Objective("Cost", lower=0)
 
-        run = Run(
-            configspace=configspace,
-            objectives=objective,
-            meta={}
+        run = BOHBRun(
+            path.stem, configspace=configspace, objectives=objective, meta={}, path=path
         )
 
         from hpbandster.core.result import logged_results_to_HBS_result
-        bohb = logged_results_to_HBS_result(base)
+
+        bohb = logged_results_to_HBS_result(str(path))
+        config_mapping = bohb.get_id2config_mapping()
 
         first_starttime = None
         for bohb_run in bohb.get_all_runs():
@@ -67,13 +59,16 @@ def get_run(self, working_dir, run_name) -> Run:
 
             cost = bohb_run.loss
             budget = bohb_run.budget
-            config = bohb_run.info["config"]
-            # Convert str to dict
-            config = json.loads(config)
+
+            if bohb_run.info is None:
+                status = "CRASHED"
+            else:
+                status = bohb_run.info["state"]
+
+            config = config_mapping[bohb_run.config_id]["config"]
 
             origin = None
             additional = {}
-            status = bohb_run.info["state"]
 
             # QUEUED, RUNNING, CRASHED, REVIEW, TERMINATED, COMPLETED, SUCCESS
             if "SUCCESS" in status or "TERMINATED" in status or "COMPLETED" in status:
@@ -98,7 +93,4 @@ def get_run(self, working_dir, run_name) -> Run:
                 additional=additional,
             )
 
-        # Save for sanity check
-        # run.save(os.path.join(base, "run"))
-
         return run
diff --git a/deepcave/runs/converters/converter.py b/deepcave/runs/converters/converter.py
deleted file mode 100644
index c2ed46cd..00000000
--- a/deepcave/runs/converters/converter.py
+++ /dev/null
@@ -1,48 +0,0 @@
-from abc import abstractmethod
-import os
-import glob
-from typing import Dict, Type, Any
-
-from deepcave.runs.run import Run
-
-
-class Converter:
-
-    @staticmethod
-    @abstractmethod
-    def name() -> str:
-        raise NotImplementedError()
-
-    @abstractmethod
-    def get_run_id(self, working_dir, run_name) -> str:
-        """
-        The id from the files in the current working_dir/run_id/*. For example, history.json could be read and hashed.
-        Idea behind: If id changed, then we have to update cached trials.
-        """
-
-        raise NotImplementedError()
-
-    @abstractmethod
-    def get_run(self, working_dir, run_name) -> Run:
-        """
-        Based on working_dir/run_name/*, return a new trials object.
-        """
-
-        raise NotImplementedError()
-
-    def get_available_run_names(self, working_dir) -> list:
-        """
-        Lists the run names in working_dir.
-        """
-
-        run_names = []
-        for run in glob.glob(os.path.join(working_dir, '*')):
-            run_name = os.path.basename(run)
-
-            try:
-                self.get_run_id(working_dir, run_name)
-                run_names.append(run_name)
-            except:
-                pass
-
-        return run_names
diff --git a/deepcave/runs/converters/deepcave.py b/deepcave/runs/converters/deepcave.py
index b5e03870..7c547c32 100644
--- a/deepcave/runs/converters/deepcave.py
+++ b/deepcave/runs/converters/deepcave.py
@@ -1,31 +1,27 @@
-import os
-import json
-import glob
-import pandas as pd
-from typing import Dict, Type, Any
+from pathlib import Path
 
-from deepcave.runs.converters.converter import Converter
 from deepcave.runs.run import Run
 from deepcave.utils.hash import file_to_hash
 
 
-class DeepCAVE(Converter):
-    @staticmethod
-    def name() -> str:
-        return "DeepCAVE"
+class DeepCAVERun(Run):
+    prefix = "DeepCAVE"
+    _initial_order = 1
 
-    def get_run_id(self, working_dir, run_name) -> str:
+    @property
+    def hash(self) -> str:
         """
-        The id from the files in the current working_dir/run_name/*. For example, history.json could be read and hashed.
+        The id from the files in the current working_dir/run_name/*. For example, history.jsonl could be read and hashed.
         Idea behind: If id changed, then we have to update cached trials.
         """
 
         # Use hash of history.json as id
-        return file_to_hash(os.path.join(working_dir, run_name, "history.jsonl"))
+        return file_to_hash(self.path / "history.jsonl")
 
-    def get_run(self, working_dir, run_name) -> Run:
+    @classmethod
+    def from_path(cls, path: Path) -> "DeepCAVERun":
         """
         Based on working_dir/run_name/*, return a new trials object.
         """
 
-        return Run(path=os.path.join(working_dir, run_name))
+        return DeepCAVERun(path.stem, path=path)
diff --git a/deepcave/runs/converters/smac.py b/deepcave/runs/converters/smac.py
index fb47dd77..cf42886f 100644
--- a/deepcave/runs/converters/smac.py
+++ b/deepcave/runs/converters/smac.py
@@ -1,43 +1,41 @@
-import os
 import json
-import glob
-import pandas as pd
+from pathlib import Path
+
 import numpy as np
-from typing import Dict, Type, Any
 
-import ConfigSpace
-from deepcave.runs.run import Status
-from deepcave.runs.converters.converter import Converter
+from deepcave.runs import Status
+from deepcave.runs.converters.deepcave import DeepCAVERun
 from deepcave.runs.objective import Objective
 from deepcave.runs.run import Run
 from deepcave.utils.hash import file_to_hash
 
 
-class SMAC(Converter):
-    @staticmethod
-    def name() -> str:
-        return "SMAC"
+class SMACRun(Run):
+    prefix = "SMAC"
+    _initial_order = 2
 
-    def get_run_id(self, working_dir, run_name) -> str:
+    @property
+    def hash(self) -> str:
         """
         The id from the files in the current working_dir/run_name/*. For example, history.json could be read and hashed.
         Idea behind: If id changed, then we have to update cached trials.
         """
 
         # Use hash of history.json as id
-        return file_to_hash(os.path.join(working_dir, run_name, "runhistory.json"))
+        return file_to_hash(self.path / "runhistory.json")
 
-    def get_run(self, working_dir, run_name) -> Run:
+    @classmethod
+    def from_path(cls, path: Path) -> "SMACRun":
         """
         Based on working_dir/run_name/*, return a new trials object.
         """
 
         # For SMAC, we create a new run object
-        base = os.path.join(working_dir, run_name)
 
         # Read configspace
         from ConfigSpace.read_and_write import json as cs_json
-        with open(os.path.join(base, 'configspace.json'), 'r') as f:
+
+        with (path / "configspace.json").open("r") as f:
             configspace = cs_json.read(f.read())
 
         # Read objectives
@@ -53,11 +51,11 @@ def get_run(self, working_dir, run_name) -> Run:
             "cutoff": "Algorithm Time Limit",
             "memory_limit": "Memory Limit",
             "wallclock_limit": "Wallclock Limit",
-            "initial_incumbent": "Initial Incumbent"
+            "initial_incumbent": "Initial Incumbent",
         }
 
         meta = {}
-        with open(os.path.join(base, "scenario.txt")) as f:
+        with (path / "scenario.txt").open() as f:
             for line in f.readlines():
                 items = line.split(" = ")
                 arg = items[0]
@@ -69,14 +67,12 @@ def get_run(self, working_dir, run_name) -> Run:
                 if arg in mapping:
                     meta[mapping[arg]] = value
 
-        run = Run(
-            configspace=configspace,
-            objectives=objective,
-            meta=meta
+        run = SMACRun(
+            path.stem, configspace=configspace, objectives=objective, meta=meta
         )
 
         # Iterate over the runhistory
-        with open(os.path.join(base, "runhistory.json")) as json_file:
+        with (path / "runhistory.json").open() as json_file:
             all_data = json.load(json_file)
             data = all_data["data"]
             config_origins = all_data["config_origins"]
@@ -84,7 +80,14 @@ def get_run(self, working_dir, run_name) -> Run:
 
         first_starttime = None
         seeds = []
-        for (config_id, instance_id, seed, budget), (cost, time, status, starttime, endtime, additional_info) in data:
+        for (config_id, instance_id, seed, budget), (
+            cost,
+            time,
+            status,
+            starttime,
+            endtime,
+            additional_info,
+        ) in data:
 
             config_id = str(config_id)
             config = configs[config_id]
@@ -137,7 +140,4 @@ def get_run(self, working_dir, run_name) -> Run:
                 additional=additional_info,
             )
 
-        # Save for sanity check
-        # run.save(os.path.join(base, "run"))
-
         return run
diff --git a/deepcave/runs/grouped_run.py b/deepcave/runs/grouped_run.py
new file mode 100644
index 00000000..1cfabf21
--- /dev/null
+++ b/deepcave/runs/grouped_run.py
@@ -0,0 +1,131 @@
+from copy import deepcopy
+from typing import List
+import numpy as np
+from deepcave.runs import AbstractRun, NotMergeableError, check_equality
+from deepcave.utils.hash import string_to_hash
+
+
+class GroupedRun(AbstractRun):
+    prefix = "group"
+
+    def __init__(self, name: str, runs: List[AbstractRun]):
+        super(GroupedRun, self).__init__(name)
+        self.runs = [run for run in runs if run is not None]  # Filter for Nones
+        self.reset()
+
+        if len(self.runs) == 0:
+            return
+
+        try:
+            attributes = check_equality(self.runs)
+            self.meta = attributes["meta"]
+            self.configspace = attributes["configspace"]
+            self.objectives = attributes["objectives"]
+            self.budgets = attributes["budgets"]
+
+            # We need new config ids
+            current_config_id = 0
+
+            # Combine runs here
+            for run in self.runs:
+                config_mapping: dict[str, str] = {}  # Maps old ones to the new ones
+
+                # Update configs + origins
+                for config_id in run.configs.keys():
+                    config = run.configs[config_id]
+                    origin = run.origins[config_id]
+
+                    for added_config_id, added_config in self.configs.items():
+                        if config == added_config:
+                            config_mapping[config_id] = added_config_id
+                            break
+
+                    if config_id not in config_mapping:
+                        self.configs[str(current_config_id)] = config
+                        self.origins[current_config_id] = origin
+                        config_mapping[config_id] = str(current_config_id)
+                        current_config_id += 1
+
+                # Update history + trial_keys
+                for trial in run.history:
+                    # Deep copy trial
+                    trial = deepcopy(trial)
+
+                    (config_id, budget) = trial.get_key()
+
+                    # Config id might have changed
+                    config_id = config_mapping[config_id]
+
+                    # Update config id
+                    trial.config_id = config_id
+
+                    # Now we add it to the history
+                    trial_key = trial.get_key()
+                    if trial_key not in self.trial_keys:
+                        self.trial_keys[trial_key] = len(self.history)
+                        self.history.append(trial)
+                    else:
+                        self.history[self.trial_keys[trial_key]] = trial
+        except:
+            raise NotMergeableError("Runs can not be merged.")
+
+    def __iter__(self):
+        for run in self.runs:
+            yield run.name
+
+    @property
+    def hash(self) -> str:
+        total_hash_str = ""
+        for run in self.runs:
+            total_hash_str += run.hash
+
+        return string_to_hash(total_hash_str)
+
+    @property
+    def run_names(self) -> list[str]:
+        return [run.name for run in self.runs]
+
+    def get_trajectory(self, *args, **kwargs):
+        # Cache costs
+        run_costs = []
+        run_times = []
+
+        # All x values on which we need y values
+        all_times = []
+
+        for run in self.runs:
+            times, costs_mean, _, _ = run.get_trajectory(*args, **kwargs)
+
+            # Cache st we don't calculate it multiple times
+            run_costs.append(costs_mean)
+            run_times.append(times)
+
+            # Add all times
+            for time in times:
+                if time not in all_times:
+                    all_times.append(time)
+
+        all_times.sort()
+
+        # Now look for corresponding y values
+        all_costs = []
+
+        for time in all_times:
+            y = []
+
+            # Iterate over all runs
+            for costs, times in zip(run_costs, run_times):
+                # Find closest x value
+                idx = min(range(len(times)), key=lambda i: abs(times[i] - time))
+                y.append(costs[idx])
+
+            all_costs.append(y)
+
+        # Make numpy arrays
+        all_costs = np.array(all_costs)
+
+        times = all_times
+        costs_mean = np.mean(all_costs, axis=1)
+        costs_std = np.std(all_costs, axis=1)
+
+        return times, list(costs_mean), list(costs_std), []
diff --git a/deepcave/runs/handler.py b/deepcave/runs/handler.py
index bd8a1f23..390234ff 100644
--- a/deepcave/runs/handler.py
+++ b/deepcave/runs/handler.py
@@ -1,213 +1,270 @@
-import os
-import json
-import hashlib
+from typing import Optional, Type
 
-from deepcave.utils.importing import auto_import_iter
-from deepcave.runs.converters.converter import Converter
-from deepcave import c, rc
-from deepcave.utils.logs import get_logger
+import time
+from pathlib import Path
 
-logger = get_logger(__name__)
+from deepcave.config import Config
+from deepcave.runs import AbstractRun, NotValidRunError
+from deepcave.runs.grouped_run import GroupedRun
+from deepcave.runs.run import Run
+from deepcave.utils.logs import get_logger
 
 
-class Handler:
+class RunHandler:
     """
     Handles the runs. Based on the meta data in the cache, automatically selects the right converter
     and switches to the right (plugin) cache.
     """
 
-    def __init__(self) -> None:
-        self.working_dir = c.get('working_dir')
-        self.run_ids = c.get('run_ids')
-        self.runs = {}
-        self.groups = {}
+    def __init__(self, config: Config, cache: "Cache", run_cache: "RunCache") -> None:
+        self.c = cache
+        self.rc = run_cache
+        # Fields set by self.update()
+        self.logger = get_logger("RunHandler")
+        self.working_dir: Optional[Path] = None
 
-        # Read from cache
-        self.update()
+        # Available converters
+        self.available_run_classes: list[Type[Run]] = config.AVAILABLE_CONVERTERS
 
-    def update(self):
-        """
-        The run caches are switched here.
-        """
+        # Internal state
+        self.runs: dict[str, Run] = {}  # run_name -> Run
+        self.groups: dict[str, GroupedRun] = {}  # group_name -> GroupedRun
 
-        working_dir = c.get('working_dir')
-        run_ids = c.get('run_ids')
-        groups = c.get('groups')
+        # Read from cache
+        self.load_from_cache()
 
-        # Set converter based on current working_dir
-        self.converter = self._find_compatible_converter(working_dir)
+    def load_from_cache(self):
+        working_dir: Path = Path(self.c.get("working_dir"))
+        selected_runs: list[str] = self.c.get("selected_run_names")  # run_name
+        groups: dict[str, list[str]] = self.c.get(
+            "groups"
+        )  # group_name -> list[run_names]
 
-        # Part where we update our runs
-        # Since it is expensive to get the current runs,
-        # only update if we have different working directory or run names
-        different = working_dir != self.working_dir or run_ids != self.run_ids
+        print(f"Resetting working directory to {working_dir}")
+        self.update_working_directory(working_dir)
 
-        if self.converter is not None and len(run_ids) > 0:
-            run_names = list(run_ids.keys())
+        print(f"Setting runs to {selected_runs}")
+        self.update_runs(selected_runs)
 
-            # Reset runs here
-            if different:
-                self.runs = {}
+        print(f"Setting groups to {groups}")
+        self.update_groups(groups)
 
-            rc.switch(working_dir, run_names)
+    def update_working_directory(
+        self, working_directory: Path, force_clear: bool = False
+    ):
+        """
+        Set working directory.
+        If it is the same as before -> Do nothing.
+        Otherwise, reset all groups and run_names and clear caches.
+        This can be forced with `force_clear`
+        """
+        if working_directory == self.working_dir and not force_clear:
+            # Same directory as current directory -> Keep everything
+            return
 
-            # We also have to clear the cache here
-            # because the data might be not accurate anymore.
-            # This is basically called on every page.
-            for run_name in run_names:
-                old_run_id = None
-                run_id = self.converter.get_run_id(working_dir, run_name)
+        # Set in runtime memory
+        self.working_dir = working_directory
+        self.update_runs([])
+        self.update_groups({})
 
-                if run_name in run_ids:
-                    old_run_id = run_ids[run_name]
+        # Set in cache
+        self.c.set("working_dir", value=str(working_directory))
 
-                # Only clear the cache if the run ids changed.
-                if run_id != old_run_id:
-                    logger.info(f"Data from {run_name} has changed ...")
+    def update_runs(self, selected_run_names: Optional[list[str]] = None):
+        """
+        Loads selected runs and update cache if files changed.
 
-                    # ... and if it is not None.
-                    if old_run_id is not None:
-                        logger.info(f"... cache was cleared.")
-                        rc[run_name].clear()
+        Raises
+        ------
+        NotValidRunError
+            If directory can not be transformed into a run, an error is thrown.
 
-                    # Update the run
-                    self.runs[run_name] = self.converter.get_run(
-                        working_dir, run_name)
+        """
+        if selected_run_names is None:
+            selected_run_names = self.c.get("selected_run_names")
+        new_runs: dict[str, Run] = {}
+
+        class_hint = None
+        for run_name in selected_run_names:
+            run = self.update_run(run_name, class_hint=class_hint)
+            if run is not None:
+                new_runs[run_name] = run
+                class_hint = run.__class__
+
+        # Save runs in memory and in cache
+        self.runs = new_runs
+        self.c.set("selected_run_names", value=self.get_run_names())
+
+    def update_run(
+        self, run_name: str, class_hint: Optional[Type[Run]] = None
+    ) -> Optional[Run]:
+        """
 
-                    logger.info(f"... run was updated.")
+        Raises
+        ------
+        NotValidRunError
+            If directory can not be transformed into a run, an error is thrown.
 
-                # Update run ids
-                run_ids[run_name] = run_id
+        """
 
-            # Really make sure all runs are set
-            for run_name in run_names:
-                if run_name not in self.runs:
-                    self.runs[run_name] = self.converter.get_run(
-                        working_dir, run_name)
+        # Try to get run from current runs
+        if run_name in self.runs:
+            run = self.runs[run_name]
+            self.rc.get_run(
+                run
+            )  # Create cache file and set name/hash. Clear cache if hash got changed
+            return run
+        else:
+            self.logger.info(f"Run {run_name} needs to be initialized")
+
+        # Load run
+        t1 = time.perf_counter()
+        run = self.get_run(run_name, class_hint=class_hint)
+        t2 = time.perf_counter()
+        self.logger.info(f"... {run_name} was loaded. (took {t2 - t1} seconds)")
+
+        # Run could not be loaded
+        if run is None:
+            self.logger.warning(f"Run {run_name} could not be loaded")
+            raise NotValidRunError()
+
+        # Add to run cache
+        self.rc.get(run)
+        return run
+
+    def update_groups(self, groups: Optional[dict[str, list[str]]] = None) -> None:
+        """
+        Loads chosen groups
 
-            # We also have to register the ids
-            self.set_run_ids(run_ids)
+        Raises
+        ------
+        NotMergeableError
+            If runs can not be merged, an error is thrown.
 
-        self.working_dir = working_dir
-        self.run_ids = run_ids
+        """
+        if groups is None:
+            groups = self.c.get("groups")
+
+        # Add groups
+        groups = {
+            name: GroupedRun(
+                name, [self.runs.get(run_name, None) for run_name in run_names]
+            )
+            for name, run_names in groups.items()
+        }
+
+        # Add groups to rc
+        for group in groups.values():
+            self.rc.get_run(
+                group
+            )  # Create cache file and set name/hash. Clear cache if hash got changed
+
+        # Save in memory
         self.groups = groups
 
-    def set_working_dir(self, working_dir=None):
-        c.set('working_dir', value=working_dir)
-
-    def set_run_names(self, run_names=[]):
-        run_ids = {}
-        for name in run_names:
-            id = None
-
-            # If id was already set, use it again
-            if name in self.run_ids:
-                id = self.run_ids[name]
-
-            run_ids[name] = id
-
-        c.set('run_ids', value=run_ids)
-
-    def set_run_ids(self, run_ids):
-        c.set('run_ids', value=run_ids)
-
-    def set_groups(self, groups):
-        c.set('groups', value=groups)
+        # Save in cache
+        groups_for_cache = {
+            name: [run.name for run in group.runs] for name, group in groups.items()
+        }
+        self.c.set("groups", value=groups_for_cache)
 
-    def get_working_dir(self):
-        self.update()
+    def get_working_dir(self) -> Path:
         return self.working_dir
 
-    def get_run_names(self):
-        self.update()
-        return list(self.run_ids.keys())
+    def get_run_names(self) -> list[str]:
+        return list(self.runs.keys())
 
-    def get_run_ids(self):
-        self.update()
-        return self.run_ids.copy()
+    def from_run_id(self, run_id: str) -> AbstractRun:
+        """
+        Required format: {prefix}:{run_name}
+        """
+        run_type, name = run_id.split(":", maxsplit=1)
+        if run_type == GroupedRun.prefix:
+            return self.groups[name]
+        else:
+            return self.runs[name]
 
-    def get_groups(self):
-        self.update()
+    def from_run_cache_id(self, run_cache_id: str) -> AbstractRun:
+        """
+        Required format: run.run_cache_id.
+        """
+        # Search in runs
+        for run in self.runs.values():
+            if run.run_cache_id == run_cache_id:
+                return run
+
+        # Search in groups
+        for group in self.groups.values():
+            if group.run_cache_id == run_cache_id:
+                return group
+
+        raise KeyError(
+            f"Could not find run with run_cache_id {run_cache_id}. "
+            f"Searched in {len(self.runs)} runs and {len(self.groups)} groups"
+        )
+
+    def get_groups(self) -> dict[str, GroupedRun]:
         return self.groups.copy()
 
-    def get_converter(self):
-        self.update()
-        return self.converter
+    def get_available_run_names(self) -> list[str]:
+        run_names = []
 
-    def get_available_run_names(self):
-        self.update()
-        if self.converter is None or self.working_dir is None:
-            return []
+        try:
+            for path in self.working_dir.iterdir():
+                run_name = path.stem
+                run_names.append(run_name)
+        except FileNotFoundError:
+            pass
 
-        return self.converter.get_available_run_names(self.working_dir)
+        return run_names
 
-    def get_runs(self):
+    def get_runs(self, include_groups=False) -> dict[str, AbstractRun]:
         """
         self.converter.get_run() might be expensive. Therefore, we cache it here, and only
         reload it, once working directory, run id or the id based on the files changed.
         """
 
-        self.update()
-        return self.runs
-
-    def _get_available_converters(self):
-        available_converters = {}
+        if include_groups:
+            # TODO: Prevent same name for runs/groups
+            runs = {}
 
-        paths = [os.path.join(os.path.dirname(__file__), 'converters/*')]
-        for _, obj in auto_import_iter("converter", paths):
-            if not issubclass(obj, Converter):
-                continue
-            # Plugin itself is a subclass, filter it out
-            if obj == Converter:
-                continue
+            # Add runs
+            for id, run in self.runs.items():
+                runs[id] = run
 
-            available_converters[obj.name()] = obj
+            # Add groups
+            for id, group in self.groups.items():
+                runs[id] = group
 
-        return available_converters
+            return runs
+        return self.runs
 
-    def _find_compatible_converter(self, working_dir):
-        """
-        All directories must be valid. Otherwise, DeepCAVE does not recognize it as compatible directory.
+    def get_run(
+        self, run_name: str, class_hint: Optional[Type[Run]] = None
+    ) -> Optional[Run]:
         """
+        Try to load run from path by using all available converters, until a sufficient class is found.
+        Try to load them in order by how many runs were already successfully converted from this class
 
-        if working_dir is None or not os.path.isdir(working_dir):
-            return None
-
-        # Find first directory
-        run_names = [name for name in os.listdir(working_dir)]
-        run_names.sort()
-
-        if len(run_names) == 0:
+        You might hint the class type if you know it with a class_hint
+        """
+        if self.working_dir is None or not self.working_dir.is_dir():
             return None
 
-        for obj in self._get_available_converters().values():
-            converter = obj()
-
-            works = True
-            for run_name in run_names:
-                if run_name == ".DS_Store":
-                    continue
-
-                try:
-                    converter.get_run(working_dir, run_name)
-                    return converter
-                except:
-                    works = False
-                    break
-
-            if works:
-                return converter
+        if class_hint is not None:
+            self.available_run_classes.remove(class_hint)
+            self.available_run_classes.insert(0, class_hint)
+
+        # Go through all converter run classes found in the order of how many runs have already been converted
+        for run_class in self.available_run_classes:
+            try:
+                run = run_class.from_path(self.working_dir / run_name)
+                self.logger.info(f"Successfully loaded {run_name} with {run_class}")
+                return run
+            except KeyboardInterrupt:
+                # Pass KeyboardInterrupt through try-except, so it can actually interrupt
+                raise
+            except:
+                pass
 
         return None
-
-    def _get_json_content(self, filename):
-        filename = os.path.join(filename)
-        with open(filename, 'r') as f:
-            data = json.load(f)
-
-        return data
-
-
-handler = Handler()
-
-__all__ = [handler]
diff --git a/deepcave/runs/objective.py b/deepcave/runs/objective.py
index 5c911cd4..1eb7e22d 100644
--- a/deepcave/runs/objective.py
+++ b/deepcave/runs/objective.py
@@ -1,8 +1,16 @@
+from typing import Optional
+
 import numpy as np
 
 
 class Objective(dict):
-    def __init__(self, name, lower=None, upper=None, optimize="lower"):
+    def __init__(
+        self,
+        name: str,
+        lower: Optional[float] = None,
+        upper: Optional[float] = None,
+        optimize="lower",
+    ):
         """
 
         Lock the lower bound if lower is not None.
@@ -40,3 +48,29 @@ def __init__(self, name, lower=None, upper=None, optimize="lower"):
         }
 
         super().__init__(data)
+
+    def merge(self, objective: "Objective"):
+        from deepcave.runs.grouped_run import NotMergeableError
+
+        attributes = ["name", "lock_lower", "lock_upper", "optimize"]
+        for attribute in attributes:
+            if self[attribute] != objective[attribute]:
+                raise NotMergeableError(f"Objective {attribute} is not mergeable.")
+
+        if self["lock_lower"] and self["lock_lower"] == objective["lock_lower"]:
+            if self["lower"] != objective["lower"]:
+                raise NotMergeableError(
+                    f"Objective {objective['name']}'s lower bound is not mergeable."
+                )
+        else:
+            if self["lower"] > objective["lower"]:
+                self["lower"] = objective["lower"]
+
+        if self["lock_upper"] and self["lock_upper"] == objective["lock_upper"]:
+            if self["upper"] != objective["upper"]:
+                raise NotMergeableError(
+                    f"Objective {objective['name']}'s upper bound is not mergeable."
+                )
+        else:
+            if self["upper"] < objective["upper"]:
+                self["upper"] = objective["upper"]
diff --git a/deepcave/runs/recorder.py b/deepcave/runs/recorder.py
index 6d0ef751..e64bddaa 100644
--- a/deepcave/runs/recorder.py
+++ b/deepcave/runs/recorder.py
@@ -1,32 +1,44 @@
+from typing import Optional, Union
 
-import glob
-import os
-import numpy as np
 import time
-from deepcave.runs.run import Status, Run
-from deepcave.utils.files import make_dirs
+from pathlib import Path
+
+import ConfigSpace
+import numpy as np
+from ConfigSpace import Configuration
+
+from deepcave.runs import Status
+from deepcave.runs.converters.deepcave import DeepCAVERun
+from deepcave.runs.run import Run
 
 
 class Recorder:
-    def __init__(self,
-                 configspace,
-                 objectives=[],
-                 meta={},
-                 save_path="logs",
-                 prefix="run",
-                 overwrite=False):
+    def __init__(
+        self,
+        configspace: ConfigSpace.ConfigurationSpace,
+        objectives=None,
+        meta=None,
+        save_path="logs",
+        prefix="run",
+        overwrite=False,
+    ):
         """
         All objectives follow the scheme the lower the better.
         If file
 
         Parameters:
-            save_path (str): Blub.
+            save_path (str):
             configspace (ConfigSpace):
             objectives (list of Objective):
             prefix: Name of the trial. If not given, trial_x will be used.
             overwrite: Uses the prefix as name and overwrites the file.
         """
+        if objectives is None:
+            objectives = []
+        if meta is None:
+            meta = {}
 
+        self.path: Path = None
         self._set_path(save_path, prefix, overwrite)
 
         # Set variables
@@ -38,10 +50,8 @@ def __init__(self,
         self.additionals = {}
 
         # Define trials container
-        self.run = Run(
-            configspace=configspace,
-            objectives=objectives,
-            meta=meta
+        self.run = DeepCAVERun(
+            self.path.stem, configspace=configspace, objectives=objectives, meta=meta
         )
 
     def __enter__(self):
@@ -50,23 +60,21 @@ def __enter__(self):
     def __exit__(self, type, value, traceback):
         pass
 
-    def _set_path(self, path, prefix="run", overwrite=False):
+    def _set_path(self, path: Union[str, Path], prefix="run", overwrite=False):
         """
         Identifies the latest run and sets the path with increased id.
         """
 
         # Make sure the word is interpreted as folder
-        if path[-1] != "/":
-            make_dirs(path + "/")
-        else:
-            make_dirs(path)
-            # Remove last slash
-            path = path[:-1]
+        path = Path(path)
+        path.mkdir(parents=True, exist_ok=True)
 
         if not overwrite:
             new_idx = 0
-            for file in glob.glob(f"{path}/{prefix}_*"):
-                idx = file.split("_")[-1]
+            for file in path.iterdir():
+                if not file.name.startswith(f"{prefix}_"):
+                    continue
+                idx = file.name.split("_")[-1]
                 if idx.isnumeric():
                     idx = int(idx)
                     if idx > new_idx:
@@ -74,17 +82,21 @@ def _set_path(self, path, prefix="run", overwrite=False):
 
             # And increase the id
             new_idx += 1
-            self.path = os.path.join(path, f"{prefix}_{new_idx}")
+            self.path = path / f"{prefix}_{new_idx}"
         else:
-            self.path = os.path.join(path, f"{prefix}")
-
-    def start(self,
-              config,
-              budget=None,
-              model=None,
-              origin=None,
-              additional={},
-              start_time=None):
+            self.path = path / f"{prefix}"
+
+    def start(
+        self,
+        config: Union[dict, Configuration],
+        budget: Optional[float] = None,
+        model=None,
+        origin=None,
+        additional: Optional[dict] = None,
+        start_time: Optional[float] = None,
+    ):
+        if additional is None:
+            additional = {}
 
         id = (config, budget)
 
@@ -99,17 +111,21 @@ def start(self,
 
         self.last_trial_id = id
 
-    def end(self,
-            costs=np.inf,
-            status=Status.SUCCESS,
-            config=None,
-            budget=np.inf,
-            additional={},
-            end_time=None):
+    def end(
+        self,
+        costs: float = np.inf,
+        status: Status = Status.SUCCESS,
+        config: Union[dict, Configuration] = None,
+        budget: float = np.inf,
+        additional: Optional[dict] = None,
+        end_time: Optional[float] = None,
+    ):
         """
         In case of multi-processing, config+budget should be passed as otherwise
         it can't be matched correctly.
         """
+        if additional is None:
+            additional = {}
 
         if config is not None:
             id = (config, budget)
@@ -134,14 +150,14 @@ def end(self,
             end_time=end_time,
             status=status,
             model=model,
-            additional=start_additional
+            additional=start_additional,
         )
 
         # Clean the dicts
-        del self.start_times[id]
-        del self.models[id]
-        del self.origins[id]
-        del self.additionals[id]
+        self.start_times.pop(id)
+        self.models.pop(id)
+        self.origins.pop(id)
+        self.additionals.pop(id)
 
         # And save the results
         self.run.save(self.path)
diff --git a/deepcave/runs/run.py b/deepcave/runs/run.py
index 34288e9c..1c6fe43f 100644
--- a/deepcave/runs/run.py
+++ b/deepcave/runs/run.py
@@ -1,32 +1,23 @@
-import os
-import numpy as np
-import jsonlines
-from enum import IntEnum
-import pandas as pd
+from abc import ABC, abstractmethod
+from typing import Any, Optional, Union
+
 import json
+from pathlib import Path
 
+import ConfigSpace
+import jsonlines
+import numpy as np
 from ConfigSpace.configuration_space import Configuration
 from ConfigSpace.read_and_write import json as cs_json
-from ConfigSpace.hyperparameters import CategoricalHyperparameter, Constant, UniformFloatHyperparameter, UniformIntegerHyperparameter
 
+from deepcave.runs import AbstractRun, Status, Trial
 from deepcave.runs.objective import Objective
 from deepcave.utils.files import make_dirs
-from deepcave.utils.logs import get_logger
-
-logger = get_logger(__name__)
-
 
-class Status(IntEnum):
-    SUCCESS = 1
-    TIMEOUT = 2
-    MEMORYOUT = 3
-    CRASHED = 4
-    ABORTED = 5
-    RUNNING = 6
 
-
-class Run:
+class Run(AbstractRun, ABC):
     """
+    TODO(dwoiwode): Docstring is outdated?
     Creates
     - meta.json
     - configspace.json
@@ -36,57 +27,68 @@ class Run:
     - models/1.blub
     """
 
-    def __init__(self,
-                 configspace=None,
-                 objectives=[],
-                 meta={},
-                 path=None):
+    prefix = "run"
+    _initial_order: int
+
+    def __init__(
+        self,
+        name: str,
+        configspace=None,
+        objectives: Union[Objective, list[Objective]] = None,
+        meta: dict[str, Any] = None,
+        path: Optional[Union[str, Path]] = None,
+    ):
         """
-        If path is given, trials are loaded from the path.
+        If path is given, runs are loaded from the path.
 
         Inputs:
             objectives (Objective or list of Objective): ...
             meta (dict): Could be `ram`, `cores`, ...
         """
+        super(Run, self).__init__(name)
+        if objectives is None:
+            objectives = []
+        if meta is None:
+            meta = {}
 
+        # Reset and load configspace/path
         self.reset()
         self.configspace = configspace
         self.path = path
         if self.path is not None:
-            return self.load()
+            self.load()
+            return
 
         if configspace is None and path is None:
             raise RuntimeError(
-                "Please provide a configspace or specify a path to load existing trials.")
+                "Please provide a configspace or specify a path to load existing trials."
+            )
 
+        # Objectives
         if not isinstance(objectives, list):
             objectives = [objectives]
 
         for objective in objectives:
             assert isinstance(objective, Objective)
 
-        self.meta = {
-            "objectives": objectives,
-            "budgets": []
-        }
+        # Meta
+        self.meta = {"objectives": objectives, "budgets": []}
         self.meta.update(meta)
 
-    def reset(self):
-        self.meta = {}
-        self.configspace = None
-        self.configs = {}
-        self.origins = {}
-        self.models = {}
-
-        self.history = []
-        self.trial_keys = {}
+    @classmethod
+    @abstractmethod
+    def from_path(cls, path: Path) -> "Run":
+        """
+        Based on a path, return a new Run object.
+        """
+        pass
 
     @property
-    def path(self):
+    def path(self) -> Optional[Path]:
         return self._path
 
     @path.setter
-    def path(self, value):
+    def path(self, value: Optional[Union[str, Path]]):
         """
         If path is changed, also change the filenames of all created files.
         """
@@ -95,38 +97,43 @@ def path(self, value):
             self._path = None
             return
 
-        if value[-1] != "/":
-            value += "/"
+        self._path = Path(value)
 
-        make_dirs(value)
-        self._path = value
+        make_dirs(self._path)
 
-        self.meta_fn = os.path.join(value, "meta.json")
-        self.configspace_fn = os.path.join(value, "configspace.json")
-        self.configs_fn = os.path.join(value, "configs.json")
-        self.origins_fn = os.path.join(value, "origins.json")
-        self.history_fn = os.path.join(value, "history.jsonl")
+        self.meta_fn = self._path / "meta.json"
+        self.configspace_fn = self._path / "configspace.json"
+        self.configs_fn = self._path / "configs.json"
+        self.origins_fn = self._path / "origins.json"
+        self.history_fn = self._path / "history.jsonl"
 
-    def exists(self):
+    def exists(self) -> bool:
         if self._path is None:
             return False
 
-        return os.path.isfile(self.meta_fn) and \
-            os.path.isfile(self.configspace_fn) and \
-            os.path.isfile(self.configs_fn) and \
-            os.path.isfile(self.origins_fn) and \
-            os.path.isfile(self.history_fn)
-
-    def add(self,
-            costs,
-            config,  # either dict or Configuration
-            budget=np.inf,
-            start_time=0.,
-            end_time=0.,
-            status=Status.SUCCESS,
-            origin=None,
-            model=None,
-            additional={}):
+        return all(
+            f.is_file()
+            for f in (
+                self.meta_fn,
+                self.configspace_fn,
+                self.configs_fn,
+                self.origins_fn,
+                self.history_fn,
+            )
+        )
+
+    def add(
+        self,
+        costs: Union[list[float], float],
+        config: Union[dict, Configuration],  # either dict or Configuration
+        budget: float = np.inf,
+        start_time: float = 0.0,
+        end_time: float = 0.0,
+        status: Status = Status.SUCCESS,
+        origin=None,
+        model=None,
+        additional: Optional[dict] = None,
+    ):
         """
 
         If combination of config and budget already exists, it will be overwritten.
@@ -134,18 +141,20 @@ def add(self,
         The cost will be calculated on the worst result later on.
 
         Inputs:
-            additional (dict): What's supported by DeepCAVE? Like `ram`, 
+            additional (dict): What's supported by DeepCAVE? Like `ram`,
             costs (float or list of floats)
         """
+        if additional is None:
+            additional = {}
 
         if not isinstance(costs, list):
             costs = [costs]
 
-        assert len(costs) == len(self.meta["objectives"])
+        assert len(costs) == len(self.get_objectives())
 
         for i in range(len(costs)):
             cost = costs[i]
-            objective = self.meta["objectives"][i]
+            objective = self.get_objectives()[i]
 
             # Update time objective here
             if objective["name"] == "time" and cost is None:
@@ -159,11 +168,11 @@ def add(self,
             # Update bounds here
             if not objective["lock_lower"]:
                 if cost < objective["lower"]:
-                    self.meta["objectives"][i]["lower"] = cost
+                    self.get_objectives()[i]["lower"] = cost
 
             if not objective["lock_upper"]:
                 if cost > objective["upper"]:
-                    self.meta["objectives"][i]["upper"] = cost
+                    self.get_objectives()[i]["upper"] = cost
 
         if isinstance(config, Configuration):
             config = config.get_dictionary()
@@ -172,8 +181,6 @@ def add(self,
             config_id = len(self.configs)
             self.configs[config_id] = config
             self.origins[config_id] = origin
-        else:
-            config_id = self.get_config_id(config)
 
         config_id = self.get_config_id(config)
         trial = Trial(
@@ -183,7 +190,7 @@ def add(self,
             start_time=np.round(start_time, 2),
             end_time=np.round(end_time, 2),
             status=status,
-            additional=additional
+            additional=additional,
         )
 
         trial_key = trial.get_key()
@@ -201,332 +208,26 @@ def add(self,
         # Update models
         self.models[trial_key] = model
 
-    def get_meta(self):
-        return self.meta
-
-    def get_objectives(self):
-        return self.meta["objectives"]
-
-    def get_objective_name(self, objective_names=None):
-        """
-        Get the cost name of given objective names.
-        Returns "Combined Cost" if multiple objective names were involved.
-        """
-
-        given_objective_names = self.get_objective_names()
-
-        if objective_names is None:
-            if len(given_objective_names) == 1:
-                return given_objective_names[0]
-        else:
-            if isinstance(objective_names, str):
-                objective_names = [objective_names]
-
-            if len(objective_names) == 1:
-                return objective_names[0]
-
-        return "Combined Cost"
-
-    def get_objective_names(self) -> list:
-        return [obj["name"] for obj in self.meta["objectives"]]
-
-    def get_config(self, id):
-        return self.configs[id]
-
-    def get_config_id(self, config: dict):
-        # Find out config id
-        for id, c in self.configs.items():
-            if c == config:
-                return id
-
-        return None
-
-    def get_configs(self, budget=None):
-        configs = []
-        for trial in self.history:
-            if budget is not None:
-                if budget != trial.budget:
-                    continue
-
-            config = self.configs[trial.config_id]
-            configs += [config]
-
-        return configs
-
-    def get_budget(self, id):
-        return self.meta["budgets"][id]
-
-    def get_budgets(self, human=False):
-        """
-        There's at least one budget with None included.
-
-        Args:
-
-        """
-        budgets = self.meta["budgets"]
-        assert len(budgets) > 0
-
-        if human:
-            readable_budgets = []
-            for b in budgets:
-                if b is None:
-                    readable_budgets += [str("None")]
-                else:
-                    readable_budgets += [str(np.round(float(b), 2))]
-
-            return readable_budgets
-
-        return budgets
-
-    def get_highest_budget(self):
-        budgets = self.meta["budgets"]
-        if len(budgets) == 0:
-            return None
-
-        return budgets[-1]
-
-    def get_costs(self, budget=None, statuses=None):
-        """
-        If no budget is given, the highest budget is chosen.
-        """
-
-        if budget is None:
-            budget = self.get_highest_budget()
-
-        results = {}
-        for trial in self.history:
-            if trial.budget is not None:
-                if trial.budget != budget:
-                    continue
-
-            if statuses is not None:
-                if trial.status not in statuses:
-                    continue
-
-            results[trial.config_id] = self._process_costs(trial.costs)
-
-        return results
-
-    def get_min_cost(self, objective_names=None, budget=None, statuses=None):
-        min_cost = np.inf
-        best_config = None
-
-        results = self.get_costs(budget, statuses)
-        for config_id, costs in results.items():
-            cost = self.calculate_cost(costs, objective_names, normalize=True)
-
-            if cost < min_cost:
-                min_cost = cost
-                best_config = self.get_config(config_id)
-
-        return min_cost, best_config
-
-    def _process_costs(self, costs):
-        """
-        Get rid of none costs.
-        """
-
-        new_costs = []
-        for idx, cost in enumerate(costs):
-            # Replace with highest cost
-            if cost is None:
-                obj = self.meta["objectives"][idx]
-                if obj["optimize"] == "lower":
-                    cost = obj["upper"]
-                else:
-                    cost = obj["lower"]
-
-            new_costs += [cost]
-
-        return new_costs
-
-    def get_trajectory(self, objective_names=None, budget=None):
-        if budget is None:
-            budget = self.get_highest_budget()
-
-        costs = []
-        ids = []
-        times = []
-
-        order = []
-        # Sort self.history by end_time
-        for id, trial in enumerate(self.history):
-            order.append((id, trial.end_time))
-
-        order.sort(key=lambda tup: tup[1])
-
-        current_cost = np.inf
-        for id, cost in order:
-            trial = self.history[id]
-            # Only consider selected/last budget
-            if trial.budget != budget:
-                continue
-
-            cost = self.calculate_cost(trial.costs, objective_names)
-            if cost < current_cost:
-                current_cost = cost
-
-                costs.append(cost)
-                times.append(trial.end_time)
-                ids.append(id)
-
-        return costs, times, ids
-
-    def calculate_cost(self, costs, objective_names=None, normalize=False):
-        """
-        Calculates cost from multiple costs.
-        Normalizes the cost first and weight every cost the same.
-        """
-
-        costs = self._process_costs(costs)
-
-        if objective_names is None:
-            objective_names = self.get_objective_names()
-
-        assert len(objective_names) > 0
-
-        # No normalization needed
-        if len(objective_names) == 1 and not normalize:
-            return costs[0]
-
-        objectives = self.meta["objectives"]
-
-        # First normalize
-        filtered_objectives = []
-        normalized_costs = []
-        for cost, objective in zip(costs, objectives):
-            if objective["name"] not in objective_names:
-                continue
-
-            a = cost - objective["lower"]
-            b = objective["upper"] - objective["lower"]
-            normalized_cost = a / b
-
-            # We optimize the lower
-            # So we need to flip the normalized cost
-            if objective["optimize"] == "upper":
-                normalized_cost = 1 - normalized_cost
-
-            normalized_costs.append(normalized_cost)
-            filtered_objectives.append(objective)
-
-        # Give the same weight to all objectives (for now)
-        objective_weights = [
-            1 / len(objectives) for _ in range(len(filtered_objectives))
-        ]
-
-        costs = [u*v for u, v in zip(normalized_costs, objective_weights)]
-        cost = np.mean(costs)
-
-        return cost
-
-    def empty(self):
-        return len(self.history) == 0
-
-    def get_encoded_configs(self,
-                            objective_names=None,
-                            budget=None,
-                            statuses=None,
-                            for_tree=False,
-                            pandas=False):
-        """
-        Args:
-            for_tree (bool): Inactives are treated differently.
-            pandas (bool): Return pandas DataFrame instead of X and Y.
-        """
-
-        X = []
-        Y = []
-
-        results = self.get_costs(budget, statuses)
-        for config_id, costs in results.items():
-
-            config = self.configs[config_id]
-            config = Configuration(self.configspace, config)
-
-            encoded = config.get_array()
-            cost = self.calculate_cost(costs, objective_names)
-
-            X.append(encoded)
-            Y.append(cost)
-
-        X = np.array(X)
-        Y = np.array(Y)
-
-        # Imputation: Easiest case is to replace all nans with -1
-        # However, since Stefan used different values for inactives
-        # we also have to use different inactives to be compatible
-        # with the random forests.
-        # https://github.com/automl/SMAC3/blob/a0c89502f240c1205f83983c8f7c904902ba416d/smac/epm/base_rf.py#L45
-
-        if not for_tree:
-            X[np.isnan(X)] = -1
-        else:
-            conditional = {}
-            impute_values = {}
-
-            for idx, hp in enumerate(self.configspace.get_hyperparameters()):
-                if idx not in conditional:
-                    parents = self.configspace.get_parents_of(hp.name)
-                    if len(parents) == 0:
-                        conditional[idx] = False
-                    else:
-                        conditional[idx] = True
-                        if isinstance(hp, CategoricalHyperparameter):
-                            impute_values[idx] = len(hp.choices)
-                        elif isinstance(hp, (UniformFloatHyperparameter, UniformIntegerHyperparameter)):
-                            impute_values[idx] = -1
-                        elif isinstance(hp, Constant):
-                            impute_values[idx] = 1
-                        else:
-                            raise ValueError
-
-                if conditional[idx] is True:
-                    nonfinite_mask = ~np.isfinite(X[:, idx])
-                    X[nonfinite_mask, idx] = impute_values[idx]
-
-        if pandas:
-            cost_column = self.get_objective_name(objective_names)
-
-            Y = Y.reshape(-1, 1)
-            data = np.concatenate((X, Y), axis=1)
-            df = pd.DataFrame(
-                data=data,
-                # Combined Cost
-                columns=[
-                    name for name in self.configspace.get_hyperparameter_names()] + [cost_column])
-
-            return df
-
-        return X, Y
-
-    def save(self, path=None):
+    def save(self, path: Optional[Union[str, Path]] = None):
         """
         If path is none, self.path will be chosen.
         """
 
-        if path is not None:
-            self.path = path
-
-        if self.path is None:
+        if path is None:
             raise RuntimeError("Please specify a path to save the trials.")
 
+        self.path = Path(path)
+
         # Save configspace
-        with open(self.configspace_fn, 'w') as f:
-            f.write(cs_json.write(self.configspace))
+        self.configspace_fn.write_text(cs_json.write(self.configspace))
 
         # Save meta data (could be changed)
-        with open(self.meta_fn, 'w') as f:
-            json.dump(self.meta, f, indent=4)
-
-        with open(self.configs_fn, 'w') as f:
-            json.dump(self.configs, f, indent=4)
-
-        with open(self.origins_fn, 'w') as f:
-            json.dump(self.origins, f, indent=4)
+        self.meta_fn.write_text(json.dumps(self.meta, indent=4))
+        self.configs_fn.write_text(json.dumps(self.configs, indent=4))
+        self.origins_fn.write_text(json.dumps(self.origins, indent=4))
 
         # Save history
-        with jsonlines.open(self.history_fn, mode='w') as f:
+        with jsonlines.open(self.history_fn, mode="w") as f:
             for trial in self.history:
                 f.write(trial)
 
@@ -534,33 +235,31 @@ def save(self, path=None):
         # to save the run.
         # Then, DeepCAVE can show direct suggestions in the select path dialog.
 
-    def load(self, path=None):
+    def load(self, path: Optional[Union[str, Path]] = None):
         self.reset()
 
+        if path is None and self.path is None:
+            raise RuntimeError("Could not load trials because path is None.")
         if path is not None:
-            self.path = path
+            self.path = Path(path)
 
         if not self.exists():
-            raise RuntimeError(
-                "Could not load trials because trials were not found.")
+            raise RuntimeError("Could not load trials because trials were not found.")
 
         # Load meta data
-        with open(self.meta_fn) as f:
-            self.meta = json.load(f)
+        self.meta = json.loads(self.meta_fn.read_text())
 
         # Load configspace
-        with open(self.configspace_fn, 'r') as f:
-            self.configspace = cs_json.read(f.read())
+        self.configspace = cs_json.read(self.configspace_fn.read_text())
 
         # Load configs
-        with open(self.configs_fn) as f:
-            configs = json.load(f)
-            self.configs = {int(k): v for k, v in configs.items()}
-            # Make sure all keys are integers
+        configs = json.loads(self.configs_fn.read_text())
+        # Make sure all keys are integers
+        self.configs = {int(k): v for k, v in configs.items()}
 
         # Load origins
-        with open(self.origins_fn) as f:
-            self.origins = json.load(f)
+        origins = json.loads(self.origins_fn.read_text())
+        self.origins = {int(k): v for k, v in origins.items()}
 
         # Load history
         with jsonlines.open(self.history_fn) as f:
@@ -575,41 +274,3 @@ def load(self, path=None):
 
         # Load models
         # TODO
-
-
-class Trial(tuple):
-    def __new__(cls, *args, **kwargs):
-
-        if len(kwargs) > 0:
-            return super(Trial, cls).__new__(cls, tuple(kwargs.values()))
-        else:
-            return super(Trial, cls).__new__(cls, tuple(args))
-
-    def __init__(self,
-                 config_id,
-                 budget,
-                 costs,
-                 start_time,
-                 end_time,
-                 status,
-                 additional):
-
-        if isinstance(status, int):
-            status = Status(status)
-
-        data = {
-            "config_id": config_id,
-            "budget": budget,
-            "costs": costs,
-            "start_time": start_time,
-            "end_time": end_time,
-            "status": status,
-            "additional": additional
-        }
-
-        # Make dict available as member variables
-        for k, v in data.items():
-            setattr(self, k, v)
-
-    def get_key(self):
-        return (self.config_id, self.budget)
diff --git a/deepcave/server.py b/deepcave/server.py
index c7a1f69b..1533e1d2 100644
--- a/deepcave/server.py
+++ b/deepcave/server.py
@@ -1,17 +1,17 @@
 import dash
 import dash_bootstrap_components as dbc
 
+from deepcave.config import Config
 
-def get_app():
+
+def get_app(config: Config = None):
+    if config is None:
+        config = Config()
     app = dash.Dash(
         __name__,
-        title='Deep CAVE',
-        update_title='',
-        external_stylesheets=[dbc.themes.BOOTSTRAP],
-        suppress_callback_exceptions=True
+        title=config.TITLE,
+        update_title="",
+        external_stylesheets=[dbc.themes.BOOTSTRAP, dbc.icons.FONT_AWESOME],
+        suppress_callback_exceptions=True,
     )
-
     return app
-
-
-app = get_app()
diff --git a/deepcave/utils/cache.py b/deepcave/utils/cache.py
index b3011880..6adbf91e 100644
--- a/deepcave/utils/cache.py
+++ b/deepcave/utils/cache.py
@@ -1,50 +1,63 @@
-import os
+from typing import Any, Optional
+
 import json
+from pathlib import Path
+
 from deepcave.utils.files import make_dirs
 
 
 class Cache:
-    def __init__(self, filename=None, defaults={}):
+    def __init__(self, filename: Optional[Path] = None, defaults=None):
         """
         Cache handles a json file. Decided not to use flask_caching
         since code is easier to change to our needs.
         """
+        self._defaults = {} if defaults is None else defaults
 
-        self._defaults = defaults
+        # Fields set by self._setup()
+        self._data = {}
+        self._file: Optional[Path] = None
+
+        # Initial setup
         self._setup(filename)
 
-    def _setup(self, filename):
+    def _setup(self, filename: Path):
         self._data = {}
-        self._filename = filename
+        self._file = filename
 
-        if filename is None:
-            return
-
-        if not os.path.exists(self._filename):
+        if filename is None or not self._file.exists():
             self.set_dict(self._defaults)
         else:
             self.read()
 
-    def switch(self, filename):
+    def switch(self, filename: Optional[Path]):
+        """Switch to a new file"""
         self._setup(filename)
 
     def read(self):
-        if not os.path.exists(self._filename):
+        """Reads content from a file and load into cache as dictionary"""
+        if not self._file.exists():
             return
 
-        with open(self._filename) as f:
+        with self._file.open("r") as f:
             self._data = json.load(f)
 
     def write(self):
-        if self._filename is None:
+        """Write content of cache into file"""
+        if self._file is None:
             return
 
-        make_dirs(self._filename)
+        self._file.parent.mkdir(exist_ok=True, parents=True)
 
-        with open(self._filename, 'w') as f:
+        with self._file.open("w") as f:
             json.dump(self._data, f, indent=4)
 
     def set(self, *keys, value):
+        """
+        Set a value from a chain of keys.
+        E.g. set("a", "b", "c", value=4) creates following dictionary:
+        {"a": {"b": {"c": 4}}}
+        """
         d = self._data
         for key in keys[:-1]:
             if key not in d:
@@ -55,11 +68,13 @@ def set(self, *keys, value):
         d[keys[-1]] = value
         self.write()
 
-    def set_dict(self, d):
+    def set_dict(self, d: dict):
+        """Updates cache to a specific value"""
         self._data.update(d)
         self.write()
 
-    def get(self, *keys):
+    def get(self, *keys) -> Optional[Any]:
+        """Retrieve value for a specific key"""
         d = self._data
         for key in keys:
             if key not in d:
@@ -69,15 +84,18 @@ def get(self, *keys):
 
         return d
 
-    def has(self, *keys):
+    def has(self, *keys) -> bool:
+        """Check whether cache has specific key"""
         d = self._data
         for key in keys:
             if key not in d:
                 return False
+            d = d[key]
 
         return True
 
     def clear(self):
+        """Clear all cache and reset to defaults"""
         self._data = {}
         self._data.update(self._defaults)
         self.write()
diff --git a/deepcave/utils/compression.py b/deepcave/utils/compression.py
index e9f6c976..f0570e98 100644
--- a/deepcave/utils/compression.py
+++ b/deepcave/utils/compression.py
@@ -1,9 +1,14 @@
+from typing import TypeVar, Union
+
 import json
+
 import numpy as np
 import pandas as pd
 
+JSON_DENSE_SEPARATORS = (",", ":")
+
 
-def serialize(data):
+def serialize(data: Union[dict, list, pd.DataFrame]) -> str:
     """
     Serialize a dataframe to a string.
     """
@@ -15,17 +20,22 @@ def default(self, obj):
             return json.JSONEncoder.default(self, obj)
 
     if isinstance(data, pd.DataFrame):
-        return json.dumps(json.loads(data.to_json()))
+        # TODO(dwoiwode): Why not just data.to_json()? Or at least make json smaller in dumps
+        return json.dumps(json.loads(data.to_json()), separators=JSON_DENSE_SEPARATORS)
+
+    return json.dumps(data, cls=Encoder, separators=JSON_DENSE_SEPARATORS)
+
 
-    return json.dumps(data, cls=Encoder)
+TYPE = TypeVar("TYPE")
 
 
-def deserialize(string, dtype=pd.DataFrame):
+def deserialize(string: str, dtype: TYPE = pd.DataFrame) -> TYPE:
     """
     Deserialize a dataframe from a string.
     """
 
     if dtype == pd.DataFrame:
+        # TODO(dwoiwode): Why not pd.read_json()?
         return pd.DataFrame.from_dict(json.loads(string))
 
     return json.loads(string)
diff --git a/deepcave/utils/dash.py b/deepcave/utils/dash.py
new file mode 100644
index 00000000..a4578a5a
--- /dev/null
+++ b/deepcave/utils/dash.py
@@ -0,0 +1,30 @@
+import dash_bootstrap_components as dbc
+
+
+def flash(message: str, category: str = "info"):
+    """
+    Flask style flash-message with Alerts
+    https://dash-bootstrap-components.opensource.faculty.ai/docs/components/alert/
+
+    Possible categories:
+        primary    -> blue
+        secondary  -> light grey
+        success    -> green
+        warning    -> yellow
+        danger     -> red
+        info       -> cyan
+        light      -> white
+        dark       -> dark gray
+    """
+    return dbc.Alert(
+        message,
+        id=f"alert_{hash(message)}",
+        is_open=True,
+        dismissable=False,
+        fade=True,
+        color=category,
+    )
+
+
+def alert(message: str):
+    return flash(message, "danger")
diff --git a/deepcave/utils/data_structures.py b/deepcave/utils/data_structures.py
index d01d3e57..fd903abd 100644
--- a/deepcave/utils/data_structures.py
+++ b/deepcave/utils/data_structures.py
@@ -1,4 +1,4 @@
-def update_dict(a, b):
+def update_dict(a: dict[str, dict], b: dict[str, dict]):
     """
     Updates a from b inplace.
     """
diff --git a/deepcave/utils/files.py b/deepcave/utils/files.py
index 1fd4f6bb..b3c887dd 100644
--- a/deepcave/utils/files.py
+++ b/deepcave/utils/files.py
@@ -1,9 +1,11 @@
-import os
+from typing import Union
 
+from pathlib import Path
 
-def make_dirs(filename):
-    # Remove last entry
-    filename_array = filename.split("/")[:-1]
-    path = "/".join(filename_array)
 
-    os.makedirs(path, exist_ok=True)
+def make_dirs(filename: Union[str, Path], parents=True):
+    path = Path(filename)
+    if path.suffix != "":  # Is file
+        path = path.parent
+
+    path.mkdir(exist_ok=True, parents=parents)
diff --git a/deepcave/utils/hash.py b/deepcave/utils/hash.py
index e55135f0..912b997e 100644
--- a/deepcave/utils/hash.py
+++ b/deepcave/utils/hash.py
@@ -1,15 +1,16 @@
 import hashlib
+from pathlib import Path
 
 
-def string_to_hash(string):
+def string_to_hash(string: str) -> str:
     hash_object = hashlib.md5(string.encode())
     return hash_object.hexdigest()
 
 
-def file_to_hash(filename):
-    with open(filename, "rb") as f:
-        hash = hashlib.md5()
+def file_to_hash(filename: Path) -> str:
+    hash = hashlib.md5()
+    with Path(filename).open("rb") as f:
         while chunk := f.read(4082):
             hash.update(chunk)
 
-        return hash.hexdigest()
+    return hash.hexdigest()
diff --git a/deepcave/utils/importing.py b/deepcave/utils/importing.py
deleted file mode 100644
index a6e8b795..00000000
--- a/deepcave/utils/importing.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import os
-import glob
-from typing import List
-from importlib import import_module
-import importlib.util
-import sys
-import inspect
-import collections
-
-from deepcave.utils.logs import get_logger
-
-
-logger = get_logger(__name__)
-
-
-def auto_import_iter(module, paths: List[str]):
-    for path in paths:
-        for f in glob.glob(path):
-            if os.path.basename(f).startswith('__'):
-                continue
-            module_name = f'{module}.' + os.path.basename(f).replace('.py', '')
-
-            if "pending" in module_name:
-                continue
-
-            # dynamic import
-            # https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path
-            try:
-                spec = importlib.util.spec_from_file_location(module_name, f)
-                foo = importlib.util.module_from_spec(spec)
-                sys.modules[spec.name] = foo
-                spec.loader.exec_module(foo)
-            except Exception as e:
-                logger.exception(
-                    f'Problem when loading file {f} as {module_name} from path {path}')
-
-            # iterate module content
-            # https://stackoverflow.com/questions/1796180/how-can-i-get-a-list-of-all-classes-within-current-module-in-python
-            # allow only class # inspect.isclass
-            for name, obj in inspect.getmembers(sys.modules[module_name], inspect.isclass):
-                yield name, obj
diff --git a/deepcave/utils/layout.py b/deepcave/utils/layout.py
index 5d24da58..3c7bb4f7 100644
--- a/deepcave/utils/layout.py
+++ b/deepcave/utils/layout.py
@@ -1,6 +1,9 @@
+from typing import Any, Optional
+
+import base64
 import io
+
 from dash import html
-import base64
 
 
 def render_table(df):
@@ -8,46 +11,50 @@ def render_table(df):
 
 
 def render_mpl_figure(fig):
+    # TODO(dwoiwode): Duplicate code (see below + ./util.py)?
     # create a virtual file which matplotlib can use to save the figure
-    buffer = BytesIO()
+    buffer = io.BytesIO()
     # save the image to memory to display in the web
-    fig.savefig(buffer, format='png', transparent=True)
+    fig.savefig(buffer, format="png", transparent=True)
     buffer.seek(0)
     # display any kind of image taken from
     # https://github.com/plotly/dash/issues/71
     encoded_image = base64.b64encode(buffer.read())
-    return html.Img(src='data:image/png;base64,{}'.format(encoded_image.decode()), className='img-fluid')
+    return html.Img(
+        src="data:image/png;base64,{}".format(encoded_image.decode()),
+        className="img-fluid",
+    )
 
 
 def display_figure(fig):
+    # TODO(dwoiwode): Duplicate code (see above + ./util.py)?
     buf = io.BytesIO()  # in-memory files
     fig.savefig(buf, format="png")  # save to the above file object
-    data = base64.b64encode(buf.getbuffer()).decode(
-        "utf8")  # encode to html elements
+    data = base64.b64encode(buf.getbuffer()).decode("utf8")  # encode to html elements
     return html.Img(src="data:image/png;base64,{}".format(data))
 
 
-def get_slider_marks(strings=None, steps=10):
+def get_slider_marks(strings: Optional[list[str]] = None, steps=10) -> dict[int, str]:
     if strings is None:
         return {0: str("None")}
 
     if len(strings) < steps:
         steps = len(strings)
 
-    marks = {}
-
     marks = {}
     for i, string in enumerate(strings):
         if i % (len(strings) / steps) == 0:
             marks[i] = str(string)
 
     # Also include the last mark
-    marks[len(strings)-1] = str(strings[-1])
+    marks[len(strings) - 1] = str(strings[-1])
 
     return marks
 
 
-def get_select_options(labels=None, values=None, binary=False):
+def get_select_options(
+    labels=None, values=None, disabled=None, binary=False
+) -> list[dict[str, Any]]:
     """
     If values are none use labels as values.
     If both are none return empty list.
@@ -60,21 +67,29 @@ def get_select_options(labels=None, values=None, binary=False):
         return []
 
     if values is None:
-        values = [l for l in labels]
+        values = labels
 
     if labels is None:
-        labels = [v for v in values]
+        labels = values
+
+    if len(labels) != len(values):
+        raise ValueError(
+            f"Labels and values have unequal length ({len(labels)} != {len(values)})"
+        )
 
     options = []
-    for l, v in zip(labels, values):
-        options.append({"label": l, "value": v})
+    for idx, (l, v) in enumerate(zip(labels, values)):
+        if disabled is not None:
+            options.append({"label": l, "value": v, "disabled": disabled[idx]})
+        else:
+            options.append({"label": l, "value": v})
 
     return options
 
 
 def get_checklist_options(labels=None, values=None, binary=False):
-    return get_select_options(labels, values, binary)
+    return get_select_options(labels=labels, values=values, binary=binary)
 
 
 def get_radio_options(labels=None, values=None, binary=False):
-    return get_select_options(labels, values, binary)
+    return get_select_options(labels=labels, values=values, binary=binary)
diff --git a/deepcave/utils/logs.py b/deepcave/utils/logs.py
index 2288f383..cfd2ccf9 100644
--- a/deepcave/utils/logs.py
+++ b/deepcave/utils/logs.py
@@ -1,14 +1,13 @@
 import logging
 import logging.config
-import os
+from pathlib import Path
 
 import yaml
 
-
-with open(os.path.join(os.path.dirname(__file__), 'logging.yml'), 'r') as stream:
+with (Path() / "logging.yml").open("r") as stream:
     config = yaml.load(stream, Loader=yaml.FullLoader)
 logging.config.dictConfig(config)
 
 
-def get_logger(logger_name):
+def get_logger(logger_name: str) -> logging.Logger:
     return logging.getLogger(logger_name)
diff --git a/deepcave/utils/run_caches.py b/deepcave/utils/run_caches.py
index 81cd64ad..2129e537 100644
--- a/deepcave/utils/run_caches.py
+++ b/deepcave/utils/run_caches.py
@@ -1,31 +1,86 @@
-import os
+from typing import Iterator, Optional, Union
+
+from deepcave.runs import AbstractRun
 from deepcave.utils.cache import Cache
-from deepcave.config import CONFIG
-from deepcave.utils.hash import string_to_hash
+from deepcave.utils.logs import get_logger
 
 
-class RunCaches(dict):
+class RunCaches:
     """
     Holds the caches for the selected runs.
     """
 
-    def switch(self, working_dir, run_names):
+    def __init__(self, config: "Config"):
+        self.cache_dir = config.CACHE_DIR / "run_cache"
+        self.logger = get_logger("RunCache")
+
+    def __getitem__(self, run: Union[str, AbstractRun]) -> Cache:
+        if isinstance(run, AbstractRun):
+            return self.get_run(run)
+        elif isinstance(run, str):  # Expect run_cache_id
+            return self.get(run)
+        else:
+            raise TypeError(
+                f"Expect Run or str (run_cache_id), but got type {type(run)} ({run})"
+            )
+
+    def __contains__(self, run: Union[AbstractRun, str]) -> bool:
+        # Resolve arguments
+        if isinstance(run, AbstractRun):
+            run_cache_id = run.run_cache_id
+        else:
+            run_cache_id = run
+
+        # Check directory for
+        for path in self.cache_dir.iterdir():
+            if path.stem == run_cache_id:
+                return True
+        return False
+
+    def __iter__(self) -> Iterator[Cache]:
+        for cache_file in self.cache_dir.iterdir():
+            yield Cache(cache_file)
+
+    def get_run(self, run: AbstractRun) -> Cache:
         """
+        Adds new files to cache. Clears cache if hash is not up-to-date
         Parameters:
-            runs (dict): A dictionary of run names and their corresponding
-                run objects.
+           run (AbstractRun): A run object, that should be cached
         """
-        self.clear()
-        data = {}
-        for run_name in run_names:
-            id = working_dir + run_name
-            hash = string_to_hash(id)
+        return self.get(run.run_cache_id, run.name, run.hash)
+
+    def get(
+        self,
+        run_cache_id: str,
+        run_name: Optional[str] = None,
+        run_hash: Optional[str] = None,
+    ) -> Cache:
+        # Create cache
+        filename = self.cache_dir / f"{run_cache_id}.json"
+        if not filename.exists():
+            self.logger.info(
+                f"Creating new cache file for {run_cache_id} at {filename.absolute().resolve()}"
+            )
+        cache = Cache(filename)
+
+        # Check whether hash is up-to-date
+        if run_hash is not None:
+            current_hash = cache.get("hash")
+            if current_hash != run_hash:
+                self.logger.info(
+                    f"Hash for {run_cache_id} has changed! ({current_hash} -> {run_hash})"
+                )
+                cache.clear()
+            cache.set("hash", value=run_hash)
 
-            filename = os.path.join(CONFIG["CACHE_DIR"], hash + ".json")
-            data[run_name] = Cache(filename)
+        # Set name after hash (otherwise might be cleared)
+        if run_name is not None:
+            cache.set("name", value=run_name)
 
-        self.update(data)
+        return cache
 
-    def clear_all(self):
-        for cache in self.values():
-            cache.clear()
+    def clear_all_caches(self):
+        """Removes all caches"""
+        files = list(self.cache_dir.iterdir())
+        for file in files:
+            file.unlink()
diff --git a/deepcave/utils/styled_plot.py b/deepcave/utils/styled_plot.py
index 68618c85..b585c846 100644
--- a/deepcave/utils/styled_plot.py
+++ b/deepcave/utils/styled_plot.py
@@ -1,11 +1,11 @@
 import matplotlib
-matplotlib.use('Agg')
-import matplotlib.pyplot as plt
 
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
 
 # IEEETrans double column standard
-FIG_WIDTH = 252.0 / 72.27 #1pt is 1/72.27 inches
-FIG_HEIGHT = FIG_WIDTH / 1.618 #golden ratio
+FIG_WIDTH = 252.0 / 72.27  # 1pt is 1/72.27 inches
+FIG_HEIGHT = FIG_WIDTH / 1.618  # golden ratio
 
 
 class StyledPlot:
@@ -15,67 +15,79 @@ class StyledPlot:
     """
 
     def __init__(self):
-        plt.style.use('seaborn')
+        plt.style.use("seaborn")
 
         # Set MatPlotLib defaults
-        plt.rcParams.update({
-            "text.usetex": True,
-            "font.family": "serif",
-            "font.serif": ["Computer Modern"],
-        })
-
-        plt.rc('xtick', labelsize=8)
-        plt.rc('ytick', labelsize=8)
-        plt.rc('axes', labelsize=12)
-        plt.rc('axes', titlesize=12)
-        plt.rc('legend', fontsize=8)
+        plt.rcParams.update(
+            {
+                "text.usetex": True,
+                "font.family": "serif",
+                "font.serif": ["Computer Modern"],
+            }
+        )
+
+        plt.rc("xtick", labelsize=8)
+        plt.rc("ytick", labelsize=8)
+        plt.rc("axes", labelsize=12)
+        plt.rc("axes", titlesize=12)
+        plt.rc("legend", fontsize=8)
 
         self.plt = plt
 
     def figure(self, cols=1, rows=1):
-        return self.plt.figure(figsize=(FIG_WIDTH*cols, FIG_HEIGHT*rows), dpi=200)
+        return self.plt.figure(figsize=(FIG_WIDTH * cols, FIG_HEIGHT * rows), dpi=200)
 
     def save_figure(self, filename):
         self.plt.savefig(filename, dpi=400, bbox_inches="tight")
         self.plt.close()
 
     def xlim(self, xmin, xmax):
-        xmin_with_margin = xmin-0.05*(xmax-xmin)
-        xmax_with_margin = xmax+0.05*(xmax-xmin)
+        xmin_with_margin = xmin - 0.05 * (xmax - xmin)
+        xmax_with_margin = xmax + 0.05 * (xmax - xmin)
         self.plt.xlim(xmin_with_margin, xmax_with_margin)
 
     def ylim(self, ymin, ymax, margin=True):
         if margin:
-            ymin_with_margin = ymin-0.05*(ymax-ymin)
-            ymax_with_margin = ymax+0.05*(ymax-ymin)
+            ymin_with_margin = ymin - 0.05 * (ymax - ymin)
+            ymax_with_margin = ymax + 0.05 * (ymax - ymin)
             self.plt.ylim(ymin_with_margin, ymax_with_margin)
         else:
             self.plt.ylim(ymin, ymax)
 
-    #def grid(self):
+    # def grid(self):
     #    pass
     #    #self.plt.grid(b=True, color='black', linestyle='--', linewidth=0.5, axis='y', zorder=0, alpha=0.5)
 
     def boxplot(self, values, positions, color, widths=0.5):
-        bp = self.plt.boxplot(values, positions=positions, patch_artist=True, widths=widths)
+        bp = self.plt.boxplot(
+            values, positions=positions, patch_artist=True, widths=widths
+        )
 
-        for box in bp['boxes']:
+        for box in bp["boxes"]:
             box.set_facecolor(color)
             box.set(linewidth=0.3)
-                
-        for whisker in bp['whiskers']:
-            whisker.set(color=color, linewidth = 0.5)
-                
-        for cap in bp['caps']:
-            cap.set(color=color, linewidth = 0.5)
-                
-        for median in bp['medians']:
-            median.set(color="black", linewidth = 0.5)
-                
-        for flier in bp['fliers']:
-            flier.set(marker ='o', markersize=3, markerfacecolor=color, linestyle='none', markeredgecolor='none', color=color, alpha = 0.5)
-
-    def legend(self, cols=1, loc='lower right', title=None, outside=False):
+
+        for whisker in bp["whiskers"]:
+            whisker.set(color=color, linewidth=0.5)
+
+        for cap in bp["caps"]:
+            cap.set(color=color, linewidth=0.5)
+
+        for median in bp["medians"]:
+            median.set(color="black", linewidth=0.5)
+
+        for flier in bp["fliers"]:
+            flier.set(
+                marker="o",
+                markersize=3,
+                markerfacecolor=color,
+                linestyle="none",
+                markeredgecolor="none",
+                color=color,
+                alpha=0.5,
+            )
+
+    def legend(self, cols=1, loc="lower right", title=None, outside=False):
         kwargs = {
             "ncol": cols,
             "columnspacing": 0.8,
@@ -86,27 +98,25 @@ def legend(self, cols=1, loc='lower right', title=None, outside=False):
             "frameon": True,
             "borderaxespad": 0.4,
             "facecolor": "white",
-            "title": title
+            "title": title,
         }
 
         if outside:
-            kwargs.update({
-                "loc": "upper left",
-                "bbox_to_anchor": (1, 1)
-            })
+            kwargs.update({"loc": "upper left", "bbox_to_anchor": (1, 1)})
 
         legend = self.plt.legend(**kwargs)
         legend.set_zorder(500)
 
         if outside:
             legend.get_frame().set_linewidth(0.0)
-            legend.get_frame().set_edgecolor('white')
+            legend.get_frame().set_edgecolor("white")
         else:
             legend.get_frame().set_linewidth(0.5)
-            legend.get_frame().set_edgecolor('gray')
+            legend.get_frame().set_edgecolor("gray")
 
     def get_color(self, id):
         import seaborn as sns
+
         pal = sns.color_palette()
         hex_codes = pal.as_hex()
 
@@ -123,4 +133,4 @@ def __getattr__(self, name):
             return self.plt.__getattribute__(name)
 
 
-plt = StyledPlot()
\ No newline at end of file
+plt = StyledPlot()
diff --git a/deepcave/utils/styled_plotty.py b/deepcave/utils/styled_plotty.py
index 68588dbf..0ea2603e 100644
--- a/deepcave/utils/styled_plotty.py
+++ b/deepcave/utils/styled_plotty.py
@@ -1,17 +1,31 @@
 import plotly.express as px
 
 
-def hex_to_rgb(hex_string):
+def hex_to_rgb(hex_string: str) -> tuple[int, int, int]:
+    """
+    Converts a hex_string to a tuple of rgb values.
+    Requires format including #, e.g.:
+    #000000
+    #ff00ff
+    """
+    if len(hex_string) != 7:
+        raise ValueError(f"Invalid length for #{hex_string}")
+
+    if any(c not in "0123456789ABCDEF" for c in hex_string.lstrip("#").upper()):
+        raise ValueError(f"Invalid character in #{hex_string}")
+
     r_hex = hex_string[1:3]
     g_hex = hex_string[3:5]
     b_hex = hex_string[5:7]
     return int(r_hex, 16), int(g_hex, 16), int(b_hex, 16)
 
 
-def get_color(id, alpha=1):
-    colors = px.colors.qualitative.Plotly
-    color = colors[id]
+def get_color(id_: int, alpha: float = 1) -> str:
+    """
+    Currently (Plotly version 5.3.1) there are 10 possible colors.
+    """
+    color = px.colors.qualitative.Plotly[id_]
 
-    R, G, B = hex_to_rgb(color)
+    r, g, b = hex_to_rgb(color)
 
-    return f"rgba({R}, {G}, {B}, {alpha})"
+    return f"rgba({r}, {g}, {b}, {alpha})"
diff --git a/deepcave/utils/util.py b/deepcave/utils/util.py
index a884c0c7..435d36d8 100644
--- a/deepcave/utils/util.py
+++ b/deepcave/utils/util.py
@@ -1,55 +1,65 @@
-import string
-from typing import Optional
+from typing import Any, Iterable, Optional, Union
+
+import base64
 import random
+import string
 from io import BytesIO
-import base64
-
-import dash_html_components as html
-import os
 
 import pandas as pd
-from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
 from ConfigSpace import ConfigurationSpace
-from ConfigSpace.hyperparameters import CategoricalHyperparameter, \
-    IntegerHyperparameter, FloatHyperparameter, OrdinalHyperparameter
+from ConfigSpace.hyperparameters import (
+    CategoricalHyperparameter,
+    FloatHyperparameter,
+    IntegerHyperparameter,
+    OrdinalHyperparameter,
+)
+from dash import html
+from matplotlib import pyplot as plt
+from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
 
 
-def get_random_string(length):
+def get_random_string(length: int) -> str:
+    if length < 0:
+        raise ValueError("Length has to be greater than 0")
     letters = string.ascii_lowercase
-    return ''.join(random.choice(letters) for i in range(length))
+    return "".join(random.choice(letters) for i in range(length))
 
 
-def matplotlib_to_html_image(fig):
+def matplotlib_to_html_image(fig: plt.Figure) -> html.Img:
+    # TODO(dwoiwode): Duplicate code (see ./layout.py)?
     # create a virtual file which matplotlib can use to save the figure
     buffer = BytesIO()
     # save the image to memory to display in the web
-    fig.savefig(buffer, format='png', transparent=True)
+    fig.savefig(buffer, format="png", transparent=True)
     buffer.seek(0)
     # display any kind of image taken from
     # https://github.com/plotly/dash/issues/71
     encoded_image = base64.b64encode(buffer.read())
-    return html.Img(src='data:image/png;base64,{}'.format(encoded_image.decode()), className='img-fluid')
+    return html.Img(
+        src=f"data:image/png;base64,{encoded_image.decode()}", className="img-fluid"
+    )
 
 
-def encode_data(data: pd.DataFrame, cs: Optional[ConfigurationSpace] = None):
-    # converts only columns with "conig." prefix
+def encode_data(
+    data: pd.DataFrame, cs: Optional[ConfigurationSpace] = None
+) -> Union[pd.DataFrame, tuple[pd.DataFrame, dict[pd.Series, pd.Series]]]:
+    # converts only columns with "config." prefix
     if cs:
         return _encode(data, cs)
     else:
         return _infer_encoding(data)
 
 
-def _infer_encoding(data: pd.DataFrame):
+def _infer_encoding(data: pd.DataFrame) -> pd.DataFrame:
     from_cols = []
     for col in data.columns:
-        if 'config.' not in col:
+        if "config." not in col:
             continue
-        if data[col].dtype == 'object':
+        if data[col].dtype == "object":
             # if not numerical convert it to numerical, via one-hot encoding
             from_cols.append(col)
     if from_cols:
-        dummies = pd.get_dummies(
-            data[from_cols], prefix=from_cols, prefix_sep='.')
+        dummies = pd.get_dummies(data[from_cols], prefix=from_cols, prefix_sep=".")
         data.drop(columns=from_cols, inplace=True)
         data = data.join(dummies)
     return data
@@ -57,13 +67,12 @@ def _infer_encoding(data: pd.DataFrame):
 
 def _encode(data: pd.DataFrame, cs: Optional):
     def _add_entry(hp_name, hp_choices, from_cols, to_cols, choices, ordinal):
-        col_name = 'config.' + hp_name
+        col_name = f"config.{hp_name}"
         from_cols.append(col_name)
         if ordinal:
             columns = [col_name]
         else:
-            columns = ['config.' + hp_name + '.' +
-                       str(col) for col in list(hp_choices)]
+            columns = [f"config.{hp_name}.{col}" for col in list(hp_choices)]
         to_cols.extend(columns)
         choices.append(list(hp_choices))
         return col_name, columns, list(hp_choices)
@@ -71,36 +80,58 @@ def _add_entry(hp_name, hp_choices, from_cols, to_cols, choices, ordinal):
     def _transform(data, from_cols, to_cols, choices, transformer_class):
         add_kwargs = {}
         if transformer_class is OneHotEncoder:
-            add_kwargs = {'sparse': False}
+            add_kwargs = {"sparse": False}
         data[to_cols] = pd.DataFrame(
-            transformer_class(categories=choices, **
-                              add_kwargs).fit_transform(data[from_cols]),
+            transformer_class(categories=choices, **add_kwargs).fit_transform(
+                data[from_cols]
+            ),
             columns=to_cols,
-            index=data.index)
+            index=data.index,
+        )
         if transformer_class is OneHotEncoder:
             data.drop(columns=from_cols, inplace=True)
         return data
 
-    org_cols = {col: col for col in data.columns if 'config.' in col}
+    org_cols = {col: col for col in data.columns if "config." in col}
     # categorical
     cat_from_cols, cat_to_cols, cat_choices = [], [], []
     ord_from_cols, ord_to_cols, ord_choices = [], [], []
     for hp in cs.get_hyperparameters():
         if isinstance(hp, CategoricalHyperparameter):
-            from_col, to_col, choices = _add_entry(hp.name, hp.choices,
-                                                   cat_from_cols, cat_to_cols, cat_choices, ordinal=False)
+            from_col, to_col, choices = _add_entry(
+                hp.name,
+                hp.choices,
+                cat_from_cols,
+                cat_to_cols,
+                cat_choices,
+                ordinal=False,
+            )
             org_cols[from_col] = to_col
         elif isinstance(hp, IntegerHyperparameter):
-            data['config.' + hp.name] = data['config.' + hp.name].astype(int)
+            data["config." + hp.name] = data["config." + hp.name].astype(int)
         elif isinstance(hp, FloatHyperparameter):
-            data['config.' + hp.name] = data['config.' + hp.name].astype(float)
+            data["config." + hp.name] = data["config." + hp.name].astype(float)
         elif isinstance(hp, OrdinalHyperparameter):
-            _add_entry(hp.name, hp.sequence, ord_from_cols,
-                       ord_to_cols, ord_choices, ordinal=True)
-
-    data = _transform(data, cat_from_cols, cat_to_cols,
-                      cat_choices, OneHotEncoder)
-    data = _transform(data, ord_from_cols, ord_to_cols,
-                      ord_choices, OrdinalEncoder)
+            _add_entry(
+                hp.name,
+                hp.sequence,
+                ord_from_cols,
+                ord_to_cols,
+                ord_choices,
+                ordinal=True,
+            )
+
+    data = _transform(data, cat_from_cols, cat_to_cols, cat_choices, OneHotEncoder)
+    data = _transform(data, ord_from_cols, ord_to_cols, ord_choices, OrdinalEncoder)
 
     return data, org_cols
+
+
+def add_prefix_to_dict(data: dict[str, Any], prefix: str) -> dict[str, Any]:
+    """Adds a prefix to every key in a dictionary"""
+    return {f"{prefix}{key}": value for key, value in data.items()}
+
+
+def add_prefix_to_list(data: Iterable[str], prefix: str) -> list[str]:
+    """Adds a prefix to every item in an iterable (e.g. a list). Returns a list"""
+    return [f"{prefix}{item}" for item in data]
diff --git a/docker-compose.yml b/docker-compose.yml
new file mode 100644
index 00000000..29df72e3
--- /dev/null
+++ b/docker-compose.yml
@@ -0,0 +1,44 @@
+version: '3'
+
+services:
+  redis:
+    image: redis:alpine
+    command: redis-server
+    ports:
+      - "6379:6379"
+    volumes:
+      - $PWD/redis-data:/var/lib/redis
+      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
+    environment:
+     - REDIS_REPLICATION_MODE=master
+    #networks:
+    #  node_net:
+    #    ipv4_address: 172.28.1.4
+
+  worker:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: python worker.py
+    volumes:
+      - .:/DeepCAVE
+    ports:
+      - "8050:8050"
+
+  server:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: python server.py
+    volumes:
+      - .:/DeepCAVE
+    ports:
+      - "8050:8050"
+
+# Networking for the Redis container
+#networks:
+#  node_net:
+#    ipam:
+#      driver: default
+#      config:
+#        - subnet: 172.28.0.0/16
\ No newline at end of file
diff --git a/environment.yml b/environment.yml
new file mode 100644
index 00000000..2091f2ed
Binary files /dev/null and b/environment.yml differ
diff --git a/examples/record/README.rst b/examples/record/README.rst
new file mode 100644
index 00000000..97af0408
--- /dev/null
+++ b/examples/record/README.rst
@@ -0,0 +1,4 @@
+Examples
+========
+
+...
\ No newline at end of file
diff --git a/examples/record_minimal_run.py b/examples/record/minimal_run.py
similarity index 56%
rename from examples/record_minimal_run.py
rename to examples/record/minimal_run.py
index 9c3c8db1..8a5ffd80 100644
--- a/examples/record_minimal_run.py
+++ b/examples/record/minimal_run.py
@@ -1,20 +1,24 @@
+"""
+Record Minimal Run
+^^^^^^^^^^^^^^^^^^
+
+...
+"""
+
+
 import numpy as np
 import ConfigSpace as CS
 from deepcave import Recorder, Objective
 
 
 configspace = CS.ConfigurationSpace(seed=0)
-alpha = CS.hyperparameters.UniformFloatHyperparameter(
-    name='alpha', lower=0, upper=1)
-beta = CS.hyperparameters.UniformFloatHyperparameter(
-    name='beta', lower=0, upper=1)
-gamma = CS.hyperparameters.UniformFloatHyperparameter(
-    name='gamma', lower=0, upper=1)
+alpha = CS.hyperparameters.UniformFloatHyperparameter(name="alpha", lower=0, upper=1)
+beta = CS.hyperparameters.Constant(name="beta", value=1)
 
-configspace.add_hyperparameters([alpha, beta, gamma])
+configspace.add_hyperparameters([alpha, beta])
 
-accuracy = Objective('accuracy', lower=0, upper=1, optimize="upper")
-time = Objective('time')
+accuracy = Objective("accuracy", lower=0, upper=1, optimize="upper")
+time = Objective("time")
 save_path = "logs/DeepCAVE"
 
 with Recorder(configspace, objectives=[accuracy, time], save_path=save_path) as r:
diff --git a/examples/record_mfis.py b/examples/record_mfis.py
deleted file mode 100644
index 2003e656..00000000
--- a/examples/record_mfis.py
+++ /dev/null
@@ -1,40 +0,0 @@
-import random
-import sys
-import time
-import ConfigSpace as CS
-from deepcave import Recorder
-
-
-if __name__ == "__main__":
-    configspace = CS.ConfigurationSpace(seed=0)
-    start = CS.hyperparameters.UniformFloatHyperparameter(
-        name='start', lower=0, upper=0.3)
-    increase = CS.hyperparameters.UniformFloatHyperparameter(
-        name='increase', lower=1, upper=1.01)
-    penalty = CS.hyperparameters.CategoricalHyperparameter(
-        name='penalty', choices=[True, False])
-    configspace.add_hyperparameters([start, increase, penalty])
-
-    with Recorder(configspace, objectives="accuracy") as r:
-        for config in configspace.sample_configuration(100):
-            for budget in [150]:
-                r.start(config, budget)
-
-                cost = config["start"]
-                cost_history = {1: cost}
-                for i in range(2, budget+1):
-                    if config["penalty"]:
-                        cost += random.uniform(0, 0.02) * \
-                            (config["increase"] / 2)
-                    else:
-                        cost += random.uniform(0, 0.02) * config["increase"]
-
-                    if cost > 1:
-                        cost = 1.
-
-                    cost_history[int(i)] = 1 - cost
-
-                r.end(costs=1 - cost,
-                      additional={"cost_history": cost_history})
-
-                time.sleep(1)
diff --git a/examples/record_openml_rf.py b/examples/record_openml_rf.py
deleted file mode 100644
index 6c404198..00000000
--- a/examples/record_openml_rf.py
+++ /dev/null
@@ -1,75 +0,0 @@
-from functools import partial
-from absl import app as flags_app
-from absl import flags
-from smac.facade.smac_hpo_facade import SMAC4HPO
-from smac.scenario.scenario import Scenario
-from hpobench.benchmarks.ml.rf_benchmark import RandomForestBenchmark
-from smac.initial_design.random_configuration_design import RandomConfigurations
-from deepcave import Recorder
-
-FLAGS = flags.FLAGS
-flags.DEFINE_integer('seed', 0, 'Seed', lower_bound=0)
-flags.DEFINE_integer('n_configs', 500, '#Configurations', lower_bound=10)
-flags.DEFINE_integer('task_id', 167149, 'Task Id', lower_bound=0)
-
-
-def eval_func(cfg, seed=0, benchmark=None):
-    result_dict = benchmark.objective_function(
-        configuration=cfg, rng=seed, fidelity={"n_estimators": 64})
-    cost = result_dict['cost']
-
-    return cost
-
-
-def main(argv):
-
-    seed = FLAGS.seed
-    task_id = FLAGS.task_id
-    n_configs = FLAGS.n_configs
-
-    benchmark = RandomForestBenchmark(task_id=task_id)
-    cs = benchmark.get_configuration_space(seed=seed)
-
-    tae = partial(eval_func, benchmark=benchmark)
-
-    #-------- START SMAC --------#
-    scenario = Scenario({
-        "run_obj": "quality",  # Optimize quality (alternatively runtime)
-        # Max number of function evaluations (the more the better)
-        "runcount-limit": n_configs,
-        "cs": cs,
-        "deterministic": True,
-    })
-
-    smac = SMAC4HPO(
-        scenario=scenario,
-        tae_runner=tae,
-        initial_design=RandomConfigurations,
-        rng=seed)
-    smac.optimize()
-
-    rh = smac.get_runhistory()
-    very_start_time = None
-
-    with Recorder(cs, prefix=f"rf_openml_{task_id}_smac_{seed}", overwrite=True) as r:
-        for (config_id, instance_id, seed, budget), (cost, time, status, starttime, endtime, additional_info) in rh.data.items():
-            config = rh.ids_config[config_id]
-            if very_start_time is None:
-                very_start_time = starttime
-
-            starttime = starttime - very_start_time
-            endtime = endtime - very_start_time
-
-            r.start(config, start_time=starttime)
-            r.end(costs=cost, end_time=endtime)
-
-    #-------- START RANDOM SEARCH --------#
-    with Recorder(cs, prefix=f"rf_openml_{task_id}_rs_{seed}", overwrite=True) as r:
-        for config in cs.sample_configuration(n_configs):
-            r.start(config)
-            cost = tae(config, seed)
-            r.end(costs=cost)
-
-
-if __name__ == "__main__":
-    flags_app.run(main)
diff --git a/deepcave/utils/logging.yml b/logging.yml
similarity index 100%
rename from deepcave/utils/logging.yml
rename to logging.yml
diff --git a/interface.png b/media/interface.png
similarity index 100%
rename from interface.png
rename to media/interface.png
diff --git a/mypy.ini b/mypy.ini
new file mode 100644
index 00000000..3ae92625
--- /dev/null
+++ b/mypy.ini
@@ -0,0 +1,8 @@
+[mypy]
+# Reports any config lines that are not recognized
+warn_unused_configs=True
+ignore_missing_imports=True
+follow_imports=skip
+disallow_untyped_defs=True
+disallow_incomplete_defs=True
+disallow_untyped_decorators=True
\ No newline at end of file
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 00000000..1119a730
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,91 @@
+# For TOML reference
+# https://learnxinyminutes.com/docs/toml/
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+minversion = "3.9"
+#addopts = "--cov=autosklearn"
+
+[tool.coverage.run]
+branch = true
+context = "deepcave"
+
+[tool.coverage.report]
+show_missing = true
+skip_covered = true
+exclude_lines = [
+    "pragma: no cover",
+    '\.\.\.',
+    "raise NotImplementedError",
+    "if TYPE_CHECKING",
+]
+
+[tool.black]
+target-version = ['py39']
+
+[tool.isort]
+py_version = "39"
+profile = "black" # Play nicely with black
+src_paths = ["deepcave", "tests"]
+known_types = ["typing", "abc"] # We put these in their own section "types"
+sections = [
+    "FUTURE",
+    "TYPES",
+    "STDLIB",
+    "THIRDPARTY",
+    "FIRSTPARTY",
+    "LOCALFOLDER",
+] # section ordering
+known_first_party = ["deepcave"]
+multi_line_output = 3 # https://pycqa.github.io/isort/docs/configuration/multi_line_output_modes.html
+
+[tool.pydocstyle]
+convention = "numpy"
+add-ignore = [ # http://www.pydocstyle.org/en/stable/error_codes.html
+    "D100", # Missing docstring in public module
+    "D101", # Missing docstring in public class
+    "D104", # Missing docstring in public package
+    "D105", # Missing docstring in magic method
+    "D203", # 1 blank line required before class docstring
+    "D205", # 1 blank line required between summary and description
+    "D210", # No whitespaces allowed surrounding docstring text
+    "D212", # Multi-line docstring summary should start at the first line
+    "D213", # Multi-line docstring summary should start at the second line
+    "D400", # First line should end with a period
+    "D401", # First line should be in imperative mood
+    "D413", # Missing blank line after last section
+    "D415", # First line should end with a period, question mark, or exclamation point
+]
+
+[tool.mypy]
+python_version = "3.9"
+
+show_error_codes = true
+
+warn_unused_configs = true # warn about unused [tool.mypy] lines
+
+follow_imports = "normal"      # Type check top level api code we use from imports
+ignore_missing_imports = false # prefer explicit ignores
+
+disallow_untyped_defs = true       # All functions must have types
+disallow_untyped_decorators = true # ... even decorators
+disallow_incomplete_defs = true    # ...all types
+
+# This is a problem with the tests of `automl_common` being distributed as a submodule
+# probably indicative that is should be a package.
+# exclude = "autosklearn/automl_common/test"
+
+# This is handled by automl_common itself in its own CI
+[[tool.mypy.overrides]]
+module = []
+ignore_errors = true
+
+# Submodules that need to be updated with mypy
+[[tool.mypy.overrides]]
+module = []
+ignore_errors = true
+
+# Packages without exported types
+[[tool.mypy.overrides]]
+module = []
+ignore_missing_imports = true
diff --git a/requirements.txt b/requirements.txt
index 241bf031..3056fea6 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,14 +1,19 @@
-ConfigSpace==0.4.20
-dash==2.0.0
-dash-bootstrap-components==1.0.0
+wheel==0.37.1
+setuptools==60.8.1
+
+absl-py==1.0.0
 jsonlines==2.0.0
 pandas==1.3.4
-plotly==5.3.1
-smac==1.1.1
-scikit-learn==1.0.1
 numpy==1.21.4
 matplotlib==3.4.3
-redis==3.5.3
+
+redis==4.1.2
 rq==1.10.0
-absl-py==1.0.0
+
+dash==2.0.0
+dash-bootstrap-components==1.0.3
+plotly==5.3.1
+
+ConfigSpace==0.4.20
+smac==1.1.1
 hpbandster==0.7.4
\ No newline at end of file
diff --git a/run.sh b/run.sh
deleted file mode 100755
index 6661fa33..00000000
--- a/run.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash
-
-trap "exit" INT TERM ERR
-trap "kill 0" EXIT
-
-# Useful for debugging
-# make clean
-
-redis-server &
-python worker.py &
-python server.py
\ No newline at end of file
diff --git a/server.py b/server.py
index 557477cb..25efb2d8 100644
--- a/server.py
+++ b/server.py
@@ -1,7 +1,12 @@
-from deepcave import app
-from deepcave.layouts.main import layout
+from deepcave import app, config
+from deepcave.layouts.main import MainLayout
 
 
-if __name__ == '__main__':
-    app.layout = layout()
-    app.run_server(debug=True)
+if __name__ == "__main__":
+    print("\n-------------STARTING SERVER-------------")
+    app.layout = MainLayout(config.PLUGINS)()
+    app.run_server(
+        debug=True,
+        # port=config.PORT,
+        # host=config.ADDRESS
+    )
diff --git a/setup.py b/setup.py
new file mode 100644
index 00000000..6819c8d3
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,80 @@
+# -*- encoding: utf-8 -*-
+import setuptools
+from deepcave import version
+
+
+def read_file(file_name):
+    with open(file_name, encoding="utf-8") as fh:
+        text = fh.read()
+    return text
+
+
+extras_require = {
+    "dev": [
+        # Tests
+        "pytest>=4.6",
+        "pytest-cov",
+        "pytest-xdist",
+        "pytest-timeout",
+        "mypy",
+        "isort",
+        "black",
+        "pydocstyle",
+        "pre-commit",
+        "flake8",
+        # Examples
+        "matplotlib",
+        "jupyter",
+        "notebook",
+        "seaborn",
+        # Docs
+        "sphinx<4.3",
+        "sphinx-gallery",
+        "sphinx_bootstrap_theme",
+        "numpydoc",
+        "sphinx_toolbox",
+        "docutils==0.16",
+    ],
+}
+
+
+setuptools.setup(
+    name="deepcave",
+    author_email="{sass, lindauer}@tnt.uni-hannover.de",
+    description="An interactive framework to visualize and analyze your AutoML process in real-time.",
+    long_description=read_file("README.md"),
+    long_description_content_type="text/markdown",
+    license="Apache-2.0",
+    url="https://www.automl.org",
+    project_urls={
+        "Documentation": "https://github.com/automl/deepcave",
+        "Source Code": "https://github.com/automl/deepcave",
+    },
+    version=version,
+    packages=setuptools.find_packages(
+        exclude=["*.tests", "*.tests.*", "tests.*", "tests"],
+    ),
+    python_requires=">=3.9",
+    install_requires=read_file("./requirements.txt").split("\n"),
+    extras_require=extras_require,
+    entry_points={
+        "console_scripts": ["deepcave = deepcave.cli:main"],
+    },
+    test_suite="pytest",
+    platforms=["Linux"],
+    classifiers=[
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Development Status :: 3 - Alpha",
+        "Natural Language :: English",
+        "Environment :: Console",
+        "Intended Audience :: Developers",
+        "Intended Audience :: Education",
+        "Intended Audience :: Science/Research",
+        "License :: OSI Approved :: Apache Software License",
+        "Operating System :: POSIX :: Linux",
+        "Topic :: Scientific/Engineering :: Artificial Intelligence",
+        "Topic :: Scientific/Engineering",
+        "Topic :: Software Development",
+    ],
+)
diff --git a/worker.py b/worker.py
index 43710c17..d724b929 100644
--- a/worker.py
+++ b/worker.py
@@ -5,6 +5,7 @@
 
 # https://github.com/tcbegley/dash-rq-demo
 if __name__ == "__main__":
+    print("\n-------------STARTING WORKER-------------")
     with Connection(queue._connection):
         w = Worker([queue._queue])
         w.work()


Test patch:
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
new file mode 100644
index 00000000..a6f43772
--- /dev/null
+++ b/.github/workflows/pytest.yml
@@ -0,0 +1,23 @@
+name: PyTest
+on: [ push ]
+jobs:
+  pytest:
+    name: PyTest
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Add conda to system path
+        run: |
+          # $CONDA is an environment variable pointing to the root of the miniconda directory
+          echo $CONDA/bin >> $GITHUB_PATH
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: Install dependencies
+        run: |
+          conda env update --file environment.yml --name base
+          pip install .
+          pip install .[dev]
+      - name: Run pytest 
+        run: pytest tests
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/test_utils.py b/tests/test_utils.py
new file mode 100644
index 00000000..9fc8c2b8
--- /dev/null
+++ b/tests/test_utils.py
@@ -0,0 +1,434 @@
+import json
+import logging
+import string
+import unittest
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import pandas as pd
+from dash import html
+
+from deepcave.utils.cache import Cache
+from deepcave.utils.compression import deserialize, serialize
+from deepcave.utils.data_structures import update_dict
+from deepcave.utils.files import make_dirs
+from deepcave.utils.hash import file_to_hash, string_to_hash
+from deepcave.utils.layout import (
+    get_checklist_options,
+    get_radio_options,
+    get_select_options,
+    get_slider_marks,
+)
+from deepcave.utils.logs import get_logger
+from deepcave.utils.styled_plotty import get_color, hex_to_rgb
+from deepcave.utils.util import (
+    add_prefix_to_dict,
+    add_prefix_to_list,
+    get_random_string,
+    matplotlib_to_html_image,
+)
+
+
+class TestCache(unittest.TestCase):
+    def test_cache_from_new_file(self):
+        cache_file = Path("tests/cache_test/cache.json")
+        cache_file.unlink(missing_ok=True)
+
+        # Load with new file
+        self.assertFalse(cache_file.exists())
+        cache = Cache(cache_file)
+
+        # Set values
+        cache.set("a", "b", "c", value=4)
+
+        # Check whether values were written to file
+        self.assertTrue(cache_file.exists())
+        with cache_file.open() as f:
+            dict_from_file = json.load(f)
+
+        self.assertIn("a", dict_from_file)
+        self.assertIn("b", dict_from_file["a"])
+        self.assertIn("c", dict_from_file["a"]["b"])
+        self.assertEqual(4, dict_from_file["a"]["b"]["c"])
+
+        # Cleanup
+        cache_file.unlink()
+
+    def test_cache_from_existing_file(self):
+        # Prepare existing file
+        cache_file = Path("tests/cache_test/cache2.json")
+        cache_file.parent.mkdir(parents=True, exist_ok=True)
+        cache_file.write_text('{"d": {"e": {"f": 32}}}')
+
+        # Load with exising file
+        self.assertTrue(cache_file.exists())
+        cache = Cache(cache_file)
+
+        # Get values
+        value = cache.get("d", "e", "f")
+        self.assertEqual(32, value)
+
+        # Cleanup
+        cache_file.unlink()
+
+    def test_cache_defaults(self):
+        cache_file = Path("tests/cache_test/cache3.json")
+        cache_file.unlink(missing_ok=True)
+
+        # Load with new file
+        self.assertFalse(cache_file.exists())
+        defaults = {"i": {"j": 4}, "k": {"l": {"m": 42}}, "v": 9}
+        cache = Cache(cache_file, defaults=defaults)
+
+        # Test get values
+        self.assertEqual(4, cache.get("i", "j"))
+        self.assertEqual(42, cache.get("k", "l", "m"))
+        self.assertEqual(9, cache.get("v"))
+
+        # Check whether values were written to file
+        self.assertTrue(cache_file.exists())
+        with cache_file.open() as f:
+            dict_from_file = json.load(f)
+
+        self.assertIn("i", dict_from_file)
+        self.assertIn("k", dict_from_file)
+        self.assertIn("v", dict_from_file)
+        self.assertIn("j", dict_from_file["i"])
+        self.assertIn("l", dict_from_file["k"])
+        self.assertIn("m", dict_from_file["k"]["l"])
+        self.assertEqual(4, dict_from_file["i"]["j"])
+        self.assertEqual(42, dict_from_file["k"]["l"]["m"])
+        self.assertEqual(9, dict_from_file["v"])
+
+        # Cleanup
+        cache_file.unlink()
+
+    def test_cache_has(self):
+        cache_file = Path("tests/cache_test/cache4.json")
+        cache_file.unlink(missing_ok=True)
+
+        # Load with new file
+        self.assertFalse(cache_file.exists())
+        defaults = {"i": {"j": 4}, "k": {"l": {"m": 42}}, "v": 9}
+        cache = Cache(cache_file, defaults=defaults)
+
+        # Test has values
+        self.assertTrue(cache.has("i"))
+        self.assertTrue(cache.has("i", "j"))
+        self.assertTrue(cache.has("k"))
+        self.assertTrue(cache.has("k", "l"))
+        self.assertTrue(cache.has("k", "l", "m"))
+        self.assertTrue(cache.has("v"))
+
+        # Cleanup
+        cache_file.unlink()
+
+    def test_cache_file_None(self):
+        """Cache should still work, even when file is None"""
+        cache = Cache(None)
+
+        cache.set(1, 2, 3, value=4)
+        self.assertEqual(4, cache.get(1, 2, 3))
+
+
+class TestCompression(unittest.TestCase):
+    def test_list_conversion(self):
+        a = [1, 2, 3, 4, 5, 6]
+        a_ser = serialize(a)
+        self.assertEqual("[1,2,3,4,5,6]", a_ser)
+        a_cycled = deserialize(a_ser, dtype=list)
+        self.assertIsInstance(a_cycled, list)
+        for i in range(len(a)):
+            self.assertEqual(a[i], a_cycled[i])
+
+    def test_dataframe_conversion(self):
+        x = [1, 2, None]
+        y = ["a", "b", "c"]
+        df = pd.DataFrame([x, y])
+
+        df_ser = serialize(df)
+        self.assertEqual(
+            "{"
+            '"0":{"0":1,"1":"a"},'
+            '"1":{"0":2,"1":"b"},'
+            '"2":{"0":null,"1":"c"}'
+            "}",
+            df_ser,
+        )
+        df_cycled = deserialize(df_ser, dtype=pd.DataFrame)
+        self.assertIsInstance(df_cycled, pd.DataFrame)
+        self.assertTrue(all((df_cycled.to_numpy() == df.to_numpy()).reshape(-1)))
+
+
+class TestDataStructures(unittest.TestCase):
+    def test_update_dict(self):
+        a = {"a": {"b": 1, "c": 2, "d": 3}}
+        b = {"a": {"b": 4, "e": 5}, "b": {"f": 6}}
+
+        update_dict(a, b)
+        self.assertIn("a", a)
+        self.assertIn("b", a)
+        a_a = a["a"]
+        self.assertEqual(4, a_a["b"])
+        self.assertEqual(2, a_a["c"])
+        self.assertEqual(3, a_a["d"])
+        self.assertEqual(5, a_a["e"])
+
+        a_b = a["b"]
+        self.assertEqual(6, a_b["f"])
+
+
+class TestFiles(unittest.TestCase):
+    def test_make_dirs(self):
+        def _test_path_procedure(argument, is_file=False):
+            path = Path(argument)
+            if is_file:
+                folder = path.parent
+            else:
+                folder = path
+            # Make sure that folder does not exist
+            if folder.exists():
+                folder.rmdir()
+
+            # Test
+            self.assertFalse(folder.exists())
+            make_dirs(argument)
+            self.assertTrue(folder.exists())
+            if is_file:
+                self.assertFalse(path.exists())
+
+            # Cleanup
+            folder.rmdir()
+
+        _test_path_procedure("tests/new_folder")
+        _test_path_procedure("tests/new_folder/file.txt", is_file=True)
+
+
+class TestHash(unittest.TestCase):
+    def test_string_to_hash(self):
+        a = string_to_hash("hello")
+        self.assertIsInstance(a, str)
+
+        b = string_to_hash("world")
+        self.assertNotEqual(a, b)
+
+        b = string_to_hash("Hello")
+        self.assertNotEqual(a, b)
+
+        b = string_to_hash("hello")
+        self.assertEqual(a, b)
+
+    def test_file_to_hash(self):
+        file = Path(__file__)
+        a = file_to_hash(file)
+        self.assertGreater(len(a), 5)
+        self.assertIsInstance(a, str)
+
+
+class TestLayout(unittest.TestCase):
+    @unittest.SkipTest
+    def test_get_slider_marks(self):
+        # TODO(dwoiwode): Currently does not work as expected?
+        default_marks = get_slider_marks()
+        self.assertIsInstance(default_marks, dict)
+        a = default_marks[0]
+        self.assertEqual("None", a)
+
+        abcde = list("ABCDE")
+        marks = get_slider_marks(abcde)
+        self.assertEqual(5, len(marks))
+        for i, c in enumerate(abcde):
+            self.assertEqual(c, marks[i])
+
+        alphabet = string.ascii_uppercase
+        marks = get_slider_marks(list(alphabet), 10)
+        self.assertEqual(10, len(marks))
+
+    def _test_get_select_options(self, method):
+        # Test empty
+        options = method(labels=None, values=None, binary=False)
+        self.assertIsInstance(options, list)
+        self.assertEqual(0, len(options))
+
+        # Test binary
+        options = method(binary=True)
+        self.assertIsInstance(options, list)
+        self.assertEqual(2, len(options))
+        no, yes = sorted(options, key=lambda o: o["value"])
+        self.assertTrue(yes["value"])
+        self.assertFalse(no["value"])
+
+        # Test copy labels
+        labels = list("ABCDEF")
+        options = method(labels)
+        self.assertEqual(len(labels), len(options))
+        for expected, actual in zip(labels, options):
+            self.assertIsInstance(actual, dict)
+            self.assertEqual(expected, actual["label"])
+            self.assertEqual(expected, actual["value"])
+
+        # Test copy values
+        values = list("12345678")
+        options = method(values=values)
+        self.assertEqual(len(values), len(options))
+        for expected, actual in zip(values, options):
+            self.assertIsInstance(actual, dict)
+            self.assertEqual(expected, actual["label"])
+            self.assertEqual(expected, actual["value"])
+
+        # Test labels + values
+        labels = list("ABCDEFGHIJ")
+        values = list("1234567890")
+        options = method(labels, values)
+        self.assertEqual(len(labels), len(options))
+        for expected_label, expected_value, actual in zip(labels, values, options):
+            self.assertIsInstance(actual, dict)
+            self.assertEqual(expected_label, actual["label"])
+            self.assertEqual(expected_value, actual["value"])
+
+        # Test unequal length labels + values
+        labels = list("ABCDEFGHIJ")
+        values = list("1234567")
+        self.assertRaises(ValueError, lambda: method(labels, values))
+
+    def test_get_select_options(self):
+        self._test_get_select_options(get_select_options)
+
+    def test_get_radio_options(self):
+        self._test_get_select_options(get_radio_options)
+
+    def test_get_checklist_options(self):
+        self._test_get_select_options(get_checklist_options)
+
+
+class TestLogger(unittest.TestCase):
+    def test_get_logger(self):
+        logger = get_logger("TestLogger")
+        self.assertIsInstance(logger, logging.Logger)
+        self.assertEqual("TestLogger", logger.name)
+
+    def test_logging_config(self):
+        mpl_logger = get_logger("matplotlib")
+        self.assertEqual(logging.INFO, mpl_logger.level)
+        self.assertFalse(mpl_logger.propagate)
+
+        plugin_logger = get_logger("src.plugins")
+        self.assertEqual(logging.DEBUG, plugin_logger.level)
+        self.assertFalse(plugin_logger.propagate)
+
+
+class TestStyledPlottly(unittest.TestCase):
+    def test_hex_to_rgb(self):
+        def assert_color(hex_code, expected_r, expected_g, expected_b):
+            r, g, b = hex_to_rgb(hex_code)
+            self.assertEqual(
+                expected_r,
+                r,
+                f"r value does not match for {hex_code} (wanted {expected_r}, got {r})",
+            )
+            self.assertEqual(
+                expected_g,
+                g,
+                f"g value does not match for {hex_code} (wanted {expected_g}, got {g})",
+            )
+            self.assertEqual(
+                expected_b,
+                b,
+                f"b value does not match for {hex_code} (wanted {expected_b}, got {b})",
+            )
+
+        assert_color("#000000", 0, 0, 0)
+        assert_color("#FFFFFF", 255, 255, 255)
+        assert_color("#ffffff", 255, 255, 255)
+        assert_color("#123456", 18, 52, 86)
+
+        self.assertRaises(ValueError, lambda: hex_to_rgb("#0g0000"))
+        self.assertRaises(ValueError, lambda: hex_to_rgb("000000"))
+
+    def test_get_color(self):
+        color_str = get_color(0)
+        self.assertEqual("rgba(99, 110, 250, 1)", color_str)
+
+        color_str = get_color(1, 0.3)
+        self.assertEqual("rgba(239, 85, 59, 0.3)", color_str)
+
+        self.assertRaises(IndexError, lambda: get_color(500))
+
+
+class TestUtil(unittest.TestCase):
+    def test_random(self):
+        # Test Length
+        a = get_random_string(10)
+        self.assertIsInstance(a, str)
+        self.assertEqual(10, len(a))
+
+        # Test random
+        b = get_random_string(10)
+        self.assertIsInstance(b, str)
+        self.assertEqual(10, len(b))
+        self.assertNotEqual(a, b)
+
+        # Test different length
+        c = get_random_string(132)
+        self.assertIsInstance(c, str)
+        self.assertEqual(132, len(c))
+
+        # Test Exception
+        self.assertRaises(ValueError, lambda: get_random_string(-1))
+
+    def test_matplotlib_to_html(self):
+        fig = plt.Figure()
+        ax = fig.gca()
+        x = [1, 2, 3, 4, 5]
+        y = [xx**2 for xx in x]
+        ax.plot(x, y)
+
+        html_img = matplotlib_to_html_image(fig)
+        self.assertIsInstance(html_img, html.Img)
+
+    @unittest.SkipTest
+    def test_encode_data(self):
+        # TODO(dwoiwode): Test with more knowledge about data structure
+        pass
+
+    @unittest.SkipTest
+    def test_encode_data_with_cs(self):
+        # TODO(dwoiwode): Test with more knowledge about data structure
+        pass
+
+    def test_add_prefix_to_dict(self):
+        a = {"a": 4, "b": 1, "c": 9}
+        a_prefixed = add_prefix_to_dict(a, "run:")
+        self.assertIn("run:a", a_prefixed)
+        self.assertIn("run:b", a_prefixed)
+        self.assertIn("run:c", a_prefixed)
+        self.assertNotIn("a", a_prefixed)
+        self.assertNotIn("b", a_prefixed)
+        self.assertNotIn("c", a_prefixed)
+        self.assertEqual(a_prefixed["run:a"], 4)
+        self.assertEqual(a_prefixed["run:b"], 1)
+        self.assertEqual(a_prefixed["run:c"], 9)
+
+        # Other prefix, same dict
+        a_prefixed = add_prefix_to_dict(a, "group:")
+        self.assertIn("group:a", a_prefixed)
+        self.assertIn("group:b", a_prefixed)
+        self.assertIn("group:c", a_prefixed)
+        self.assertNotIn("a", a_prefixed)
+        self.assertNotIn("b", a_prefixed)
+        self.assertNotIn("c", a_prefixed)
+        self.assertEqual(a_prefixed["group:a"], 4)
+        self.assertEqual(a_prefixed["group:b"], 1)
+        self.assertEqual(a_prefixed["group:c"], 9)
+
+    def test_add_prefix_to_list(self):
+        a = ["a", "b", "Hello", "World"]
+        a_prefixed = add_prefix_to_list(a, "run:")
+        self.assertIsInstance(a_prefixed, list)
+        self.assertEqual(4, len(a_prefixed))
+        self.assertEqual("run:a", a_prefixed[0])
+        self.assertEqual("run:b", a_prefixed[1])
+        self.assertEqual("run:Hello", a_prefixed[2])
+
+        a_prefixed = add_prefix_to_list(a, "job:")
+        self.assertEqual("job:World", a_prefixed[3])