Repository: openml/automlbenchmark (language: python)

Issue:
Not enough precision in predict_duration in results.csv
We are [rounding](https://github.com/openml/automlbenchmark/blob/aee089164e8236e581672732c8bdc5ee6393d0ac/amlb/results.py#L172) to the first decimal for `predict_duration` so a lot of the results are getting rounded to 0.

Here, `"{:.1f}".format`, we need .3g instead of .1f.
fix for mlr3automl
makes PR #424 obsolete
Fix/merge back v2
previous attempt was squash-merged, apparently deleting the common node between stable-v2 and master
https://download.mlplan.org/ does not have mlplan.zip
The discussion started in #463 but MlPlan is still failing.

Should it be this [file](https://download.mlplan.org/ailibs-0.9.0.tar.gz) (see https://download.mlplan.org/)?

Github actions is failing at this `mlplanweka` task.
EC2 AMIs: Outdated
The current EC2 AMis used in AMLB are very old (I believe they were created in ~2019):

```
  ec2:
    regions:                             #
      us-east-1:
        ami: ami-0ac019f4fcb7cb7e6
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      us-west-1:
        ami: ami-063aa838bd7631e0b
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      eu-west-1:
        ami: ami-00035f41c82244dab
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      eu-central-1:
        ami: ami-0bdf93799014acdc4
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
```

This leads to them having security vulnerabilities and thus are not able to be SSH'ed into when using an AWS account with proper security controls.

I would strongly recommend these AMI's be updated to 2023 AMIs.

I have tested the following us-east-1 AMI which runs successfully on AutoGluon:

```
    # Note: Added a new updated region image, old images no longer responded to SSH and didn't work correctly
    regions:                             #
      us-east-1:
        ami: ami-09cd747c78a9add63
        description: Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2023-02-07
```

I found this updated AMI by:

1. Going to [https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstances:](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstances:)
2. Clicking 'Browse more AMIs'
3. Under 'Quickstart AMIs', selecting 'Ubuntu Server 20.04 LTS (HVM), SSD Volume Type'
4. In the AMI from catalog view, you can see the AMI ID:
```
Amazon Machine Image (AMI)

ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230207
ami-09cd747c78a9add63
```
-> 
```
ami-09cd747c78a9add63
```

We can see that this is a recently created AMI:

```
Published

2023-02-07T22:43:00.000Z
```

We should be able to repeat this process for every region we want to support, and then update the `config.yaml` with the new AMIs.

Note: I updated to 20.04 from 18.04, but there is also an updated 18.04 AMI image too in case using Ubuntu 18.04 is preferred over 20.04.

```
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20230131
ami-0263e4deb427da90e
```
```
Published

2023-01-31T16:36:13.000Z
```


Discussion / hints:

Fix patch:
diff --git a/.github/scripts/version-set.sh b/.github/scripts/version-set.sh
index 734f64144..3f67a039e 100755
--- a/.github/scripts/version-set.sh
+++ b/.github/scripts/version-set.sh
@@ -3,7 +3,7 @@ VERSION="$1"
 PUSH=${2:-"no-push"}
 ADD_COMMIT=${3:-"add-commit"}
 
-VERSION_FILE='app/__version__.py'
+VERSION_FILE='amlb/__version__.py'
 VERSION_PLACEHOLDER='__version__ =\s+"(.*)"$'
 DEV_PLACEHOLDER='_dev_version =\s+"(.*)"$'
 
diff --git a/.github/workflows/run_all_frameworks.yml b/.github/workflows/run_all_frameworks.yml
index 542df1d48..b11ce4629 100644
--- a/.github/workflows/run_all_frameworks.yml
+++ b/.github/workflows/run_all_frameworks.yml
@@ -15,7 +15,7 @@ jobs:
       skip_baseline: ${{ steps.find-required-tests.outputs.skip_baseline }}
       skip_evaluation: ${{ steps.find-required-tests.outputs.skip_evaluation }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: pull base branch
         run: |
           git fetch --unshallow origin $GITHUB_BASE_REF
@@ -52,7 +52,7 @@ jobs:
           skip_evaluation=0
           if [[ $is_common -eq 0 ]];
           then
-            FRAMEWORKS='["autogluon", "autosklearn", "autoxgboost", "gama", "h2oautoml", "mlplanweka", "mlr3automl", "tpot", "tunedrandomforest"]'
+            FRAMEWORKS='["autogluon", "autosklearn", "gama", "h2oautoml", "mlplanweka", "mlr3automl", "tpot", "tunedrandomforest"]'
             TASKS='["iris", "kc2", "cholesterol"]'
             BENCHMARK='["test"]'
           else
@@ -90,17 +90,17 @@ jobs:
         task: [iris, kc2, cholesterol]
       fail-fast:  false
     steps:
-      - uses: actions/checkout@v2
-      - name: Setup Python 3.8
-        uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
         with:
-          python-version: '3.8'
+          python-version: '3.9'
       - name: Create venv
         run: python -m venv venv
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         id: cache
         with:
-          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.8/site-packages
+          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
           key: pip-v3-${{ hashFiles('**/requirements.txt') }}
           restore-keys: |
             pip-v3-
@@ -126,24 +126,24 @@ jobs:
 
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.9]
         framework: ${{ fromJson(needs.detect_changes.outputs.frameworks) }}
         task: ${{ fromJson(needs.detect_changes.outputs.tasks) }}
         benchmark: ${{ fromJson(needs.detect_changes.outputs.benchmark) }}
       fail-fast:  true  # not sure about this one, but considering the big workload it might be nicer
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Setup Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
     - name: Create venv
       run: python -m venv venv
-    - uses: actions/cache@v2
+    - uses: actions/cache@v3
       id: cache
       with:
-        path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.8/site-packages
+        path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
         key: pip-v3-${{ hashFiles('**/requirements.txt') }}
         restore-keys: |
           pip-v3-
@@ -152,8 +152,10 @@ jobs:
       run: |
         source venv/bin/activate
         python -m pip install --upgrade pip
-        pip install -r requirements.txt
+        python -m pip install -r requirements.txt
     - name: Run ${{ matrix.framework }} on ${{ matrix.task }}
       run: |
         source venv/bin/activate
         python runbenchmark.py ${{ matrix.framework }} ${{ matrix.benchmark }} test -f 0 -t ${{ matrix.task }} -e
+      env:
+        GITHUB_PAT: ${{ secrets.PUBLIC_ACCESS_GITHUB_PAT }}
diff --git a/.github/workflows/runbenchmark.yml b/.github/workflows/runbenchmark.yml
new file mode 100644
index 000000000..20bde27af
--- /dev/null
+++ b/.github/workflows/runbenchmark.yml
@@ -0,0 +1,38 @@
+name: Run `runbenchmark.py`
+
+on:
+  workflow_dispatch:
+    inputs:
+      python_version:
+        default: '3.9'
+      os:
+        default: 'ubuntu-latest'
+      framework:
+        default: 'randomforest'
+      options:
+        default: ''
+
+jobs:
+  run_benchmark:
+    runs-on: ${{ inputs.os }}
+    steps:
+      - name: Runs on ${{ inputs.os }}
+        run: echo Preparing to benchmark ${{ inputs.framework }} ${{ inputs.options }}
+      - name: Checkout ${{ github.ref_name }}
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.ref_name }}
+      - name: Install Py${{ inputs.python_version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ inputs.python_version }}
+      - name: Install benchmark dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -r requirements.txt
+      - name: Install ${{ inputs.framework }}
+        run: python runbenchmark.py ${{ inputs.framework }} -s only
+        env:
+          GITHUB_PAT: ${{ secrets.PUBLIC_ACCESS_GITHUB_PAT }}
+      - name: Benchmark ${{ inputs.framework }}
+        run: python runbenchmark.py ${{ inputs.framework }} ${{ inputs.options }}
diff --git a/.github/workflows/versioning-reset.yml b/.github/workflows/versioning-reset.yml
index 63fbf6424..8900a377e 100644
--- a/.github/workflows/versioning-reset.yml
+++ b/.github/workflows/versioning-reset.yml
@@ -7,7 +7,7 @@ on:
       - main
       - master
     paths:
-      - 'app/__version__.py'
+      - 'amlb/__version__.py'
 
   workflow_dispatch:
 
diff --git a/.gitignore b/.gitignore
index e3cad6b3e..bc9c76adc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,6 +16,7 @@ venv/
 .idea/
 *.iml
 *.swp
+launch.json
 
 # tmp files
 .ipynb_checkpoints/
@@ -47,3 +48,4 @@ Singularityfile
 #whitelist
 !reports/**/*.csv
 !reports/**/*.png
+!docs/img/**/*.png
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index b4bc78a5d..000000000
--- a/.travis.yml
+++ /dev/null
@@ -1,12 +0,0 @@
-language: python
-python:
-  - "3.7.9"
-
-before_install:
-  - python -m pip install --upgrade pip
-install:
-  - pip install -r requirements.txt
-  - pip install -r requirements-dev.txt
-
-script:
-  - python -m pytest -m "not stress"
diff --git a/amlb/__version__.py b/amlb/__version__.py
index 46945d24b..8c165df2f 100644
--- a/amlb/__version__.py
+++ b/amlb/__version__.py
@@ -1,3 +1,3 @@
 # major.minor.patch[-label]
 _dev_version = "dev"
-__version__ = "2.0.6"
+__version__ = "dev"
diff --git a/amlb/benchmark.py b/amlb/benchmark.py
index 4fbc93a3f..572dc91a3 100644
--- a/amlb/benchmark.py
+++ b/amlb/benchmark.py
@@ -7,6 +7,7 @@
 - run the jobs.
 - collect and save results.
 """
+import time
 from copy import copy
 from enum import Enum
 from importlib import import_module, invalidate_caches
@@ -24,8 +25,9 @@
 from .datautils import read_csv
 from .resources import get as rget, config as rconfig, output_dirs as routput_dirs
 from .results import ErrorResult, Scoreboard, TaskResult
-from .utils import Namespace as ns, OSMonitoring, as_list, datetime_iso, flatten, json_dump, lazy_property, profile, repr_def, \
-    run_cmd, run_script, signal_handler, str2bool, str_sanitize, system_cores, system_memory_mb, system_volume_mb, touch
+from .utils import Namespace as ns, OSMonitoring, as_list, datetime_iso, file_lock, flatten, json_dump, \
+    lazy_property, profile, repr_def, run_cmd, run_script, signal_handler, str2bool, str_sanitize, \
+    system_cores, system_memory_mb, system_volume_mb, touch
 
 
 log = logging.getLogger(__name__)
@@ -209,22 +211,21 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
             results = self._run_jobs(jobs)
             log.info(f"Processing results for {self.sid}")
             log.debug(results)
-            if tasks is None:
-                scoreboard = self._process_results(results)
-            else:
-                for task_def in task_defs:
-                    task_results = filter(lambda res: res.result is not None and res.result.task == task_def.name, results)
-                    scoreboard = self._process_results(task_results, task_name=task_def.name)
-            return scoreboard
+
+            if not rconfig().results.incremental_save:
+                self._process_results(results)
+            return self._results_summary()
         finally:
             self.cleanup()
 
     def _create_job_runner(self, jobs):
+        on_new_result = self._process_results if rconfig().results.incremental_save else None
         if self.parallel_jobs == 1:
-            return SimpleJobRunner(jobs)
+            return SimpleJobRunner(jobs, on_new_result=on_new_result)
         else:
-            # return ThreadPoolExecutorJobRunner(jobs, self.parallel_jobs)
-            return MultiThreadingJobRunner(jobs, self.parallel_jobs,
+            return MultiThreadingJobRunner(jobs,
+                                           on_new_result=on_new_result,
+                                           parallel_jobs=self.parallel_jobs,
                                            delay_secs=rconfig().job_scheduler.delay_between_jobs,
                                            done_async=True)
 
@@ -254,10 +255,6 @@ def on_interrupt(*_):
             pass
         finally:
             results = self.job_runner.results
-
-        for res in results:
-            if res.result is not None and math.isnan(res.result.duration):
-                res.result.duration = res.duration
         return results
 
     def _benchmark_tasks(self):
@@ -327,34 +324,45 @@ def _skip_job(self, task_def, fold):
 
         return False
 
-    def _process_results(self, results, task_name=None):
+    def _process_results(self, results):
+        if not isinstance(results, list):
+            results = [results]
         scores = list(filter(None, flatten([res.result for res in results])))
         if len(scores) == 0:
             return None
 
-        board = (Scoreboard(scores,
-                            framework_name=self.framework_name,
-                            task_name=task_name,
-                            scores_dir=self.output_dirs.scores) if task_name
-                 else Scoreboard(scores,
-                                 framework_name=self.framework_name,
-                                 benchmark_name=self.benchmark_name,
-                                 scores_dir=self.output_dirs.scores))
+        for res in results:
+            if math.isnan(res.result.duration):
+                res.result.duration = res.duration
 
-        if rconfig().results.save:
-            self._save(board)
-
-        log.info("Summing up scores for current run:\n%s",
-                 board.as_printable_data_frame(verbosity=2).dropna(how='all', axis='columns').to_string(index=False))
-        return board.as_data_frame()
+        board = Scoreboard(scores, scores_dir=self.output_dirs.scores)
+        self._save(board)
+        return board
 
     def _save(self, board):
         board.save(append=True)
-        self._append(board)
-
-    def _append(self, board):
-        Scoreboard.all().append(board).save()
-        Scoreboard.all(rconfig().output_dir).append(board).save()
+        self._save_global(board)
+
+    def _save_global(self, board):
+        # Scoreboard.all().append(board).save()
+        if rconfig().results.global_save:
+            global_board = Scoreboard.all(rconfig().output_dir, autoload=False)
+            dest_path = global_board.path
+            timeout = rconfig().results.global_lock_timeout
+            try:
+                with file_lock(dest_path, timeout=timeout):
+                    global_board.load().append(board).save()
+            except TimeoutError:
+                log.exception("Failed to acquire the lock on `%s` after %ss: "
+                              "the partial board `%s` could not be appended to `%s`",
+                              dest_path, timeout, board.path, dest_path)
+
+    def _results_summary(self, scoreboard=None):
+        board = scoreboard or Scoreboard.all(self.output_dirs.scores)
+        results = board.as_printable_data_frame(verbosity=2)
+        log.info("Summing up scores for current run:\n%s",
+                 results.dropna(how='all', axis='columns').to_string(index=False))
+        return board.as_data_frame()
 
     @lazy_property
     def output_dirs(self):
@@ -371,14 +379,16 @@ def _is_task_enabled(task_def):
 
 class TaskConfig:
 
-    def __init__(self, name, fold, metrics, seed,
+    def __init__(self, name, openml_task_id, test_server, fold, metrics, seed,
                  max_runtime_seconds, cores, max_mem_size_mb, min_vol_size_mb,
-                 input_dir, output_dir):
+                 input_dir, output_dir, tag, command, git_info, measure_inference_time: bool = False):
         self.framework = None
         self.framework_params = None
         self.framework_version = None
         self.type = None
         self.name = name
+        self.openml_task_id = openml_task_id
+        self.test_server = test_server
         self.fold = fold
         self.metrics = [metrics] if isinstance(metrics, str) else metrics
         self.seed = seed
@@ -389,19 +399,29 @@ def __init__(self, name, fold, metrics, seed,
         self.input_dir = input_dir
         self.output_dir = output_dir
         self.output_predictions_file = os.path.join(output_dir, "predictions.csv")
+        self.tag = tag
+        self.command = command
+        self.git_info = git_info
+        self.measure_inference_time = measure_inference_time
         self.ext = ns()  # used if frameworks require extra config points
 
     def __setattr__(self, name, value):
         if name == 'metrics':
             self.metric = value[0] if isinstance(value, list) else value
         elif name == 'max_runtime_seconds':
-            self.job_timeout_seconds = min(value * 2,
-                                           value + rconfig().benchmarks.overhead_time_seconds)
+            inference_time_extension = 0
+            if rconfig().inference_time_measurements.enabled:
+                inference_time_extension = rconfig().inference_time_measurements.additional_job_time
+            self.job_timeout_seconds = min(value * 2 + inference_time_extension,
+                                           value + rconfig().benchmarks.overhead_time_seconds + inference_time_extension)
         super().__setattr__(name, value)
 
     def __json__(self):
         return self.__dict__
 
+    def __repr__(self):
+        return repr_def(self)
+
     def estimate_system_params(self):
         on_unfulfilled = rconfig().benchmarks.on_unfulfilled_constraint
         mode = re.split(r"\W+", rconfig().run_mode, maxsplit=1)[0]
@@ -457,6 +477,7 @@ def __init__(self, benchmark: Benchmark, task_def, fold):
         self.fold = fold
         self.task_config = TaskConfig(
             name=task_def.name,
+            openml_task_id=task_def.openml_task_id,
             fold=fold,
             metrics=task_def.metric,
             seed=rget().seed(fold),
@@ -466,6 +487,11 @@ def __init__(self, benchmark: Benchmark, task_def, fold):
             min_vol_size_mb=task_def.min_vol_size_mb,
             input_dir=rconfig().input_dir,
             output_dir=benchmark.output_dirs.session,
+            test_server=rget().config.test_server,
+            tag=rget().config.__dict__.get("tag"),
+            command=rget().config.command,
+            git_info=rget().git_info,
+            measure_inference_time=rconfig().inference_time_measurements.enabled,
         )
         # allowing to override some task parameters through command line, e.g.: -Xt.max_runtime_seconds=60
         if rconfig()['t'] is not None:
@@ -486,7 +512,9 @@ def load_data(self):
             # TODO
             raise NotImplementedError("OpenML datasets without task_id are not supported yet.")
         elif hasattr(self._task_def, 'dataset'):
-            self._dataset = Benchmark.data_loader.load(DataSourceType.file, dataset=self._task_def.dataset, fold=self.fold)
+            dataset_name_and_config = copy(self._task_def.dataset)
+            dataset_name_and_config.name = self._task_def.name
+            self._dataset = Benchmark.data_loader.load(DataSourceType.file, dataset=dataset_name_and_config, fold=self.fold)
         else:
             raise ValueError("Tasks should have one property among [openml_task_id, openml_dataset_id, dataset].")
 
@@ -519,7 +547,12 @@ def run(self):
                              predictions_dir=self.benchmark.output_dirs.predictions)
         framework_def = self.benchmark.framework_def
         task_config = copy(self.task_config)
-        task_config.type = 'regression' if self._dataset.type == DatasetType.regression else 'classification'
+        if self._dataset.type == DatasetType.regression:
+            task_config.type = 'regression'
+        elif self._dataset.type == DatasetType.timeseries:
+            task_config.type = 'timeseries'
+        else:
+            task_config.type = 'classification'
         task_config.type_ = self._dataset.type.name
         task_config.framework = self.benchmark.framework_name
         task_config.framework_params = framework_def.params
@@ -538,7 +571,7 @@ def run(self):
 
         result = meta_result = None
         try:
-            log.info("Running task %s on framework %s with config:\n%s", task_config.name, self.benchmark.framework_name, repr_def(task_config))
+            log.info("Running task %s on framework %s with config:\n%s", task_config.name, self.benchmark.framework_name, task_config)
             json_dump(task_config, task_config.output_metadata_file, style='pretty')
             meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
         except Exception as e:
@@ -549,4 +582,3 @@ def run(self):
         finally:
             self._dataset.release()
         return results.compute_score(result=result, meta_result=meta_result)
-
diff --git a/amlb/benchmarks/openml.py b/amlb/benchmarks/openml.py
index 33d70d8fb..78aa01242 100644
--- a/amlb/benchmarks/openml.py
+++ b/amlb/benchmarks/openml.py
@@ -16,7 +16,7 @@ def is_openml_benchmark(benchmark: str) -> bool:
         supported_types = ['s', 't']
 
         if oml_id.isdecimal():
-            return domain == "openml" and oml_type in supported_types
+            return domain in ("openml", "test.openml") and oml_type in supported_types
     return False
 
 
@@ -26,6 +26,10 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
     path = None  # benchmark file does not exist on disk
     name = benchmark  # name is later passed as cli input again for containers, it needs to remain parsable
 
+    if domain == "test.openml":
+        log.debug("Setting openml server to the test server.")
+        openml.config.server = "https://test.openml.org/api/v1/xml"
+
     if openml.config.retry_policy != "robot":
         log.debug(
             "Setting openml retry_policy from '%s' to 'robot'." % openml.config.retry_policy)
@@ -38,7 +42,8 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
         data = openml.datasets.get_dataset(t.dataset_id, download_data=False, download_qualities=False)
         tasks = [Namespace(name=str_sanitize(data.name),
                            description=data.description,
-                           openml_task_id=t.id)]
+                           openml_task_id=t.id,
+                           id="{}.org/t/{}".format(domain, t.id))]
     elif oml_type == 's':
         log.info("Loading openml suite %s.", oml_id)
         suite = openml.study.get_suite(oml_id)
@@ -50,7 +55,8 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
         for tid, did in zip(suite.tasks, suite.data):
             tasks.append(Namespace(name=str_sanitize(datasets.loc[did]['name']),
                                    description=f"{openml.config.server.replace('/api/v1/xml', '')}/d/{did}",
-                                   openml_task_id=tid))
+                                   openml_task_id=tid,
+                                   id="{}.org/t/{}".format(domain, tid)))
     else:
         raise ValueError(f"The oml_type is {oml_type} but must be 's' or 't'")
     return name, path, tasks
diff --git a/amlb/data.py b/amlb/data.py
index b329df27b..acca17841 100644
--- a/amlb/data.py
+++ b/amlb/data.py
@@ -43,7 +43,7 @@ def __init__(self, index, name, data_type, values=None, has_missing_values=False
         self.index = index
         self.name = name
         self.data_type = data_type.lower() if data_type is not None else None
-        self.values = self.normalize(values).tolist() if values is not None else None
+        self.values = values
         self.has_missing_values = has_missing_values
         self.is_target = is_target
         # print(self)
@@ -80,8 +80,16 @@ def one_hot_encoder(self):
     def normalize(self, arr):
         return np.char.lower(np.char.strip(np.asarray(arr).astype(str)))
 
+    @property
+    def values(self):
+        return self._values
+
+    @values.setter
+    def values(self, values):
+        self._values = self.normalize(values).tolist() if values is not None else None
+
     def __repr__(self):
-        return repr_def(self)
+        return repr_def(self, 'all')
 
 
 class Datasplit(ABC):
@@ -134,8 +142,7 @@ def y(self) -> DF:
     @lazy_property
     @profile(logger=log)
     def data_enc(self) -> AM:
-        data = np.where(self.data.notna(), self.data.values, None)
-        encoded_cols = [f.label_encoder.transform(data[:, f.index]) for f in self.dataset.features]
+        encoded_cols = [f.label_encoder.transform(self.data.iloc[:, f.index]) for f in self.dataset.features]
         # optimize mem usage : frameworks use either raw data or encoded ones,
         # so we can clear the cached raw data once they've been encoded
         self.release(['data', 'X', 'y'])
@@ -165,6 +172,7 @@ class DatasetType(Enum):
     binary = 1
     multiclass = 2
     regression = 3
+    timeseries = 4
 
 
 class Dataset(ABC):
diff --git a/amlb/datasets/file.py b/amlb/datasets/file.py
index bcb941ab2..ee92d3b1a 100644
--- a/amlb/datasets/file.py
+++ b/amlb/datasets/file.py
@@ -13,10 +13,10 @@
 from ..data import Dataset, DatasetType, Datasplit, Feature
 from ..datautils import read_csv, to_data_frame
 from ..resources import config as rconfig
-from ..utils import Namespace as ns, as_list, lazy_property, list_all_files, memoize, path_from_split, profile, split_path
-
-from .fileutils import download_file, is_archive, is_valid_url, unarchive_file, url_exists
+from ..utils import Namespace as ns, as_list, lazy_property, list_all_files, memoize, path_from_split, profile, repr_def, split_path
 
+from .fileutils import is_archive, is_valid_url, unarchive_file, get_file_handler
+from copy import deepcopy
 
 log = logging.getLogger(__name__)
 
@@ -33,7 +33,7 @@ def __init__(self, cache_dir=None):
     def load(self, dataset, fold=0):
         dataset = dataset if isinstance(dataset, ns) else ns(path=dataset)
         log.debug("Loading dataset %s", dataset)
-        paths = self._extract_train_test_paths(dataset.path if 'path' in dataset else dataset, fold=fold)
+        paths = self._extract_train_test_paths(dataset.path if 'path' in dataset else dataset, fold=fold, name=dataset['name'] if 'name' in dataset else None)
         assert fold < len(paths['train']), f"No training dataset available for fold {fold} among dataset files {paths['train']}"
         # seed = rget().seed(fold)
         # if len(paths['test']) == 0:
@@ -51,21 +51,28 @@ def load(self, dataset, fold=0):
         if ext == '.arff':
             return ArffDataset(train_path, test_path, target=target, features=features, type=type_)
         elif ext == '.csv':
-            return CsvDataset(train_path, test_path, target=target, features=features, type=type_)
+            if type_ is not None and DatasetType[type_] == DatasetType.timeseries and dataset['timestamp_column'] is None:
+                log.warning("Warning: For timeseries task setting undefined timestamp column to `timestamp`.")
+                dataset = deepcopy(dataset)
+                dataset['timestamp_column'] = "timestamp"
+            csv_dataset = CsvDataset(train_path, test_path, target=target, features=features, type=type_, timestamp_column=dataset['timestamp_column'] if 'timestamp_column' in dataset else None)
+            if csv_dataset.type == DatasetType.timeseries:
+                csv_dataset = self.extend_dataset_with_timeseries_config(csv_dataset, dataset)
+            return csv_dataset
         else:
             raise ValueError(f"Unsupported file type: {ext}")
 
-    def _extract_train_test_paths(self, dataset, fold=None):
+    def _extract_train_test_paths(self, dataset, fold=None, name=None):
         if isinstance(dataset, (tuple, list)):
             assert len(dataset) % 2 == 0, "dataset list must contain an even number of paths: [train_0, test_0, train_1, test_1, ...]."
             return self._extract_train_test_paths(ns(train=[p for i, p in enumerate(dataset) if i % 2 == 0],
                                                      test=[p for i, p in enumerate(dataset) if i % 2 == 1]),
-                                                  fold=fold)
+                                                  fold=fold, name=name)
         elif isinstance(dataset, ns):
-            return dict(train=[self._extract_train_test_paths(p)['train'][0]
+            return dict(train=[self._extract_train_test_paths(p, name=name)['train'][0]
                                if i == fold else None
                                for i, p in enumerate(as_list(dataset.train))],
-                        test=[self._extract_train_test_paths(p)['train'][0]
+                        test=[self._extract_train_test_paths(p, name=name)['train'][0]
                               if i == fold else None
                               for i, p in enumerate(as_list(dataset.test))])
         else:
@@ -116,14 +123,55 @@ def _extract_train_test_paths(self, dataset, fold=None):
                 assert len(paths) > 0, f"No dataset file found in {dataset}: they should follow the naming xxxx_train.ext, xxxx_test.ext or xxxx_train_0.ext, xxxx_test_0.ext, xxxx_train_1.ext, ..."
                 return paths
         elif is_valid_url(dataset):
-            cached_file = os.path.join(self._cache_dir, os.path.basename(dataset))
+            if name is None:
+                cached_file = os.path.join(self._cache_dir, os.path.basename(dataset))
+            else:
+                cached_file = os.path.join(self._cache_dir, name, os.path.basename(dataset))
             if not os.path.exists(cached_file):  # don't download if previously done
-                assert url_exists(dataset), f"Invalid path/url: {dataset}"
-                download_file(dataset, cached_file)
+                handler = get_file_handler(dataset)
+                assert handler.exists(dataset), f"Invalid path/url: {dataset}"
+                handler.download(dataset, dest_path=cached_file)
             return self._extract_train_test_paths(cached_file)
         else:
             raise ValueError(f"Invalid dataset description: {dataset}")
 
+    def __repr__(self):
+        return repr_def(self)
+
+
+    def extend_dataset_with_timeseries_config(self, dataset, dataset_config):
+        dataset = deepcopy(dataset)
+        dataset_config = deepcopy(dataset_config)
+        if dataset_config['id_column'] is None:
+            log.warning("Warning: For timeseries task setting undefined `id_column` to `item_id`.")
+            dataset_config['id_column'] = "item_id"
+        if dataset_config['forecast_range_in_steps'] is None:
+            log.warning("Warning: For timeseries task setting undefined `forecast_range_in_steps` to `1`.")
+            dataset_config['forecast_range_in_steps'] = "1"
+
+        dataset.timestamp_column=dataset_config['timestamp_column']
+        dataset.id_column=dataset_config['id_column']
+        dataset.forecast_range_in_steps=int(dataset_config['forecast_range_in_steps'])
+
+        train_seqs_lengths = dataset.train.X.groupby(dataset.id_column).count()
+        test_seqs_lengths = dataset.test.X.groupby(dataset.id_column).count()
+        forecast_range_in_steps_mean_diff_train_test = int((test_seqs_lengths - train_seqs_lengths).mean())
+        forecast_range_in_steps_max_min_train_test = int(min(int(test_seqs_lengths.min()), int(train_seqs_lengths.min()))) - 1
+        if not dataset.forecast_range_in_steps == forecast_range_in_steps_mean_diff_train_test:
+            msg = f"Warning: Forecast range {dataset.forecast_range_in_steps}, does not equal mean difference between test and train sequence lengths {forecast_range_in_steps_mean_diff_train_test}."
+            log.warning(msg)
+        if not (test_seqs_lengths - train_seqs_lengths).var().item() == 0.:
+            msg = f"Error: Not all sequences of train and test set have same sequence length difference."
+            raise ValueError(msg)
+        if dataset.forecast_range_in_steps > forecast_range_in_steps_mean_diff_train_test:
+            msg = f"Error: Forecast range {dataset.forecast_range_in_steps} longer than difference between test and train sequence lengths {forecast_range_in_steps_mean_diff_train_test}."
+            raise ValueError(msg)
+        if dataset.forecast_range_in_steps > forecast_range_in_steps_max_min_train_test:
+            msg = f"Error: Forecast range {dataset.forecast_range_in_steps} longer than minimum sequence length + 1, {forecast_range_in_steps_max_min_train_test}."
+            raise ValueError(msg)
+        return dataset
+
+
 
 class FileDataset(Dataset):
 
@@ -165,6 +213,9 @@ def _get_metadata(self, prop):
         meta = self._train.load_metadata()
         return meta[prop]
 
+    def __repr__(self):
+        return repr_def(self, 'all')
+
 
 class FileDatasplit(Datasplit):
 
@@ -211,11 +262,17 @@ def _set_feature_as_target(self, target: Feature):
         ds_type = self.dataset._type
         if ds_type and DatasetType[ds_type] in [DatasetType.binary, DatasetType.multiclass]:
             if not target.is_categorical():
-                log.warning("Forcing target column %s as 'category' for classification problems: was originally detected as '%s'.",
+                log.warning("Forcing target column `%s` as 'category' for classification problems: was originally detected as '%s'.",
                             target.name, target.data_type)
-                # target.data_type = 'category'
+                self._convert_to_categorical(target)
         target.is_target = True
 
+    def _convert_to_categorical(self, feature: Feature):
+        feature.data_type = 'category'
+
+    def __repr__(self):
+        return repr_def(self, 'all')
+
 
 class ArffDataset(FileDataset):
 
@@ -289,35 +346,44 @@ def release(self, properties=None):
 class CsvDataset(FileDataset):
 
     def __init__(self, train_path, test_path,
-                 target=None, features=None, type=None):
+                 target=None, features=None, type=None, timestamp_column=None):
         # todo: handle auto-split (if test_path is None): requires loading the training set, split, save
-        super().__init__(CsvDatasplit(self, train_path), CsvDatasplit(self, test_path),
+        super().__init__(None, None,
                          target=target, features=features, type=type)
+        self._train = CsvDatasplit(self, train_path, timestamp_column=timestamp_column)
+        self._test = CsvDatasplit(self, test_path, timestamp_column=timestamp_column)
         self._dtypes = None
 
 
 class CsvDatasplit(FileDatasplit):
 
-    def __init__(self, dataset, path):
+    def __init__(self, dataset, path, timestamp_column=None):
         super().__init__(dataset, format='csv', path=path)
         self._ds = None
+        self.timestamp_column = timestamp_column
 
     def _ensure_loaded(self):
         if self._ds is None:
             if self.dataset._dtypes is None:
-                df = read_csv(self.path)
+                df = read_csv(self.path, timestamp_column=self.timestamp_column)
                 # df = df.convert_dtypes()
                 dt_conversions = {name: 'category'
                                   for name, dtype in zip(df.dtypes.index, df.dtypes.values)
-                                  if pat.is_string_dtype(dtype) or pat.is_object_dtype(dtype)}
+                                  if pat.is_string_dtype(dtype)
+                                  or pat.is_object_dtype(dtype)
+                                  or (name == self.dataset._target
+                                      and self.dataset._type is not None
+                                      and DatasetType[self.dataset._type] in [DatasetType.binary, DatasetType.multiclass])
+                                  }
                 # we could be a bit more clever in the future and convert 'string' to category iff len(distinct values) << nrows
                 if dt_conversions:
                     df = df.astype(dt_conversions, copy=False)
 
                 self._ds = df
                 self.dataset._dtypes = self._ds.dtypes
+
             else:
-                self._ds = read_csv(self.path, dtype=self.dataset._dtypes.to_dict())
+                self._ds = read_csv(self.path, dtype=self.dataset._dtypes.to_dict(), timestamp_column=self.timestamp_column)
 
     @profile(logger=log)
     def load_metadata(self):
@@ -328,7 +394,7 @@ def load_metadata(self):
                                       else 'number' if pat.is_numeric_dtype(dt)
                                       else 'category' if pat.is_categorical_dtype(dt)
                                       else 'string' if pat.is_string_dtype(dt)
-                                      # else 'datetime' if pat.is_datetime64_dtype(dt)
+                                      else 'datetime' if pat.is_datetime64_dtype(dt)
                                       else 'object')
         features = [Feature(i, col, to_feature_type(dtypes[i]))
                     for i, col in enumerate(self._ds.columns)]
@@ -336,8 +402,9 @@ def load_metadata(self):
         for f in features:
             col = self._ds.iloc[:, f.index]
             f.has_missing_values = col.hasnans
+            # f.dtype = self._ds.dtypes[f.name]
             if f.is_categorical():
-                f.values = sorted(self._ds.dtypes[f.name].categories.values)
+                f.values = self._unique_values(f.name)
 
         target = self._find_target_feature(features)
         self._set_feature_as_target(target)
@@ -358,6 +425,11 @@ def release(self, properties=None):
         super().release(properties)
         self._ds = None
 
+    def _unique_values(self, col_name: str):
+        dt = self._ds.dtypes[col_name]
+        return sorted(dt.categories.values if hasattr(dt, 'categories')
+                      else self._ds[col_name].unique())
+
 
 class FileConverter:
     format = None
@@ -383,22 +455,24 @@ class ArffConverter(FileConverter):
 
     def _write_file(self, df, path):
         name = split_path(path).basename
+        description = f"Arff dataset file generated by automlbenchmark from {name}."
+        attributes = [(c,
+                       ('INTEGER' if pat.is_integer_dtype(dt)
+                        else 'REAL' if pat.is_float_dtype(dt)
+                        else 'NUMERIC' if pat.is_numeric_dtype(dt)
+                        # numeric categories need to be str: https://github.com/renatopp/liac-arff/issues/126
+                        else [str(cat) for cat in sorted(dt.categories.values)] if pat.is_categorical_dtype(dt)
+                        else 'STRING'
+                        ))
+                      for c, dt in zip(df.columns, df.dtypes)]
+        arff_data = dict(
+            description=description,
+            relation=name,
+            attributes=attributes,
+            data=df.values
+        )
         with open(path, 'w') as file:
-            description = f"Arff dataset file generated by automlbenchmark from {name}."
-            attributes = [(c,
-                           ('INTEGER' if pat.is_integer_dtype(dt)
-                            else 'REAL' if pat.is_float_dtype(dt)
-                            else 'NUMERIC' if pat.is_numeric_dtype(dt)
-                            else sorted(dt.categories.values) if pat.is_categorical_dtype(dt)
-                            else 'STRING'
-                            ))
-                          for c, dt in zip(df.columns, df.dtypes)]
-            arff.dump(dict(
-                description=description,
-                relation=name,
-                attributes=attributes,
-                data=df.values
-            ), file)
+            arff.dump(arff_data, file)
 
 
 class CsvConverter(FileConverter):
diff --git a/amlb/datasets/fileutils.py b/amlb/datasets/fileutils.py
index bce8cbc4f..f1b9eece9 100644
--- a/amlb/datasets/fileutils.py
+++ b/amlb/datasets/fileutils.py
@@ -2,39 +2,85 @@
 import os
 import shutil
 import tarfile
+import boto3
+from botocore.errorfactory import ClientError
 from urllib.error import URLError
 from urllib.parse import urlparse
-from urllib.request import Request, urlopen, urlretrieve
+from urllib.request import Request, urlopen
 import zipfile
 
 from ..utils import touch
 
 log = logging.getLogger(__name__)
 
-SUPPORTED_SCHEMES = ("http", "https")
 
+class FileHandler:
+    def exists(self, url): pass
+    def download(self, url, dest_path): pass
+
+
+class HttpHandler(FileHandler):
+    def exists(self, url):
+        head_req = Request(url, method='HEAD')
+        try:
+            with urlopen(head_req) as test:
+                return test.status == 200
+        except URLError as e:
+            log.error(f"Cannot access url %s: %s", url, e)
+            return False
+    
+    def download(self, url, dest_path):
+        touch(dest_path)
+        with urlopen(url) as resp, open(dest_path, 'wb') as dest:
+            shutil.copyfileobj(resp, dest)
+
+
+class S3Handler(FileHandler):
+    def exists(self, url):
+        s3 = boto3.client('s3')
+        bucket, key = self._s3_path_to_bucket_prefix(url)
+        try:
+            s3.head_object(Bucket=bucket, Key=key)
+            return True
+        except ClientError as e:
+            log.error(f"Cannot access url %s: %s", url, e)
+            return False
+        
+    def download(self, url, dest_path):
+        touch(dest_path)
+        s3 = boto3.resource('s3')
+        bucket, key = self._s3_path_to_bucket_prefix(url)
+        try:
+            s3.Bucket(bucket).download_file(key, dest_path)
+        except ClientError as e:
+            if e.response['Error']['Code'] == "404":
+                log.error("The object does not exist.")
+            else:
+                raise
+        
+    def _s3_path_to_bucket_prefix(self, s3_path):
+        s3_path_cleaned = s3_path.split('://', 1)[1]
+        bucket, prefix = s3_path_cleaned.split('/', 1)
+        return bucket, prefix
 
-def is_valid_url(url):
-    return urlparse(url).scheme in SUPPORTED_SCHEMES
 
+scheme_handlers = dict(
+    http=HttpHandler(),
+    https=HttpHandler(),
+    s3=S3Handler(),
+    s3a=S3Handler(),
+    s3n=S3Handler(),
+)
 
-def url_exists(url):
-    if not is_valid_url(url):
-        return False
-    head_req = Request(url, method='HEAD')
-    try:
-        with urlopen(head_req) as test:
-            return test.status == 200
-    except URLError as e:
-        log.error(f"Cannot access url %s: %s", url, e)
-        return False
+SUPPORTED_SCHEMES = list(scheme_handlers.keys())
 
 
-def download_file(url, dest_path):
-    touch(dest_path)
-    # urlretrieve(url, filename=dest_path)
-    with urlopen(url) as resp, open(dest_path, 'wb') as dest:
-        shutil.copyfileobj(resp, dest)
+def get_file_handler(url):
+    return scheme_handlers[urlparse(url).scheme]
+
+
+def is_valid_url(url):
+    return urlparse(url).scheme in SUPPORTED_SCHEMES
 
 
 def is_archive(path):
@@ -50,6 +96,24 @@ def unarchive_file(path, dest_folder=None):
             zf.extractall(path=dest_folder)
     elif tarfile.is_tarfile(path):
         with tarfile.open(path) as tf:
-            tf.extractall(path=dest_folder)
+            def is_within_directory(directory, target):
+                
+                abs_directory = os.path.abspath(directory)
+                abs_target = os.path.abspath(target)
+            
+                prefix = os.path.commonprefix([abs_directory, abs_target])
+                
+                return prefix == abs_directory
+            
+            def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
+            
+                for member in tar.getmembers():
+                    member_path = os.path.join(path, member.name)
+                    if not is_within_directory(path, member_path):
+                        raise Exception("Attempted Path Traversal in Tar File")
+            
+                tar.extractall(path, members, numeric_owner=numeric_owner) 
+                
+            
+            safe_extract(tf, path=dest_folder)
     return dest
-
diff --git a/amlb/datasets/openml.py b/amlb/datasets/openml.py
index d037252d8..657f49f99 100644
--- a/amlb/datasets/openml.py
+++ b/amlb/datasets/openml.py
@@ -2,21 +2,24 @@
 **openml** module implements the abstractions defined in **data** module
 to expose `OpenML<https://www.openml.org>`_ datasets.
 """
+import pathlib
 from abc import abstractmethod
 import copy
 import functools
 import logging
 import os
 import re
-from typing import Generic, Tuple, TypeVar, Union
+from typing import Generic, Tuple, TypeVar
 
 import arff
+import pandas as pd
 import pandas.api.types as pat
 import openml as oml
 import xmltodict
 
 from ..data import AM, DF, Dataset, DatasetType, Datasplit, Feature
-from ..resources import config as rconfig
+from ..datautils import impute_array
+from ..resources import config as rconfig, get as rget
 from ..utils import as_list, lazy_property, path_from_split, profile, split_path, unsparsify
 
 
@@ -66,6 +69,15 @@ def __init__(self, oml_task: oml.OpenMLTask, oml_dataset: oml.OpenMLDataset, fol
         self.fold = fold
         self._train = None
         self._test = None
+        self._nrows = None
+
+
+    @property
+    def nrows(self) -> int:
+        if self._nrows is None:
+            self._nrows = len(self._load_full_data(fmt='dataframe'))
+        return self._nrows
+
 
     @lazy_property
     def type(self):
@@ -92,6 +104,80 @@ def test(self):
         self._ensure_split_created()
         return self._test
 
+    def inference_subsample_files(self, fmt: str, with_labels: bool = False, scikit_safe: bool = False) -> list[Tuple[int, str]]:
+        """Generates n subsamples of size k from the test dataset in `fmt` data format.
+
+        We measure the inference time of the models for various batch sizes
+        (number of rows). We generate config.inference_time_measurements.repeats
+        subsamples for each of the config.inference_time_measurements.batch_sizes.
+
+        These subsamples are stored to file in the `fmt` format (parquet, arff, or csv).
+        The function returns a list of tuples of (batch size, file path).
+
+        Iff `with_labels` is true, the target column will be included in the split file.
+        Iff `scikit_safe` is true, categorical values are encoded and missing values
+        are imputed.
+        """
+        seed = rget().seed(self.fold)
+        batch_sizes = [
+            batch_size for batch_size in rconfig().inference_time_measurements.batch_sizes
+            if not (batch_size > self.nrows and rconfig().inference_time_measurements.limit_by_dataset_size)
+        ]
+        return [
+            (n, str(self._inference_subsample(fmt=fmt, n=n, seed=seed + i, with_labels=with_labels, scikit_safe=scikit_safe)))
+            for n in batch_sizes
+            for i, _ in enumerate(range(rconfig().inference_time_measurements.repeats))
+        ]
+
+    @profile(logger=log)
+    def _inference_subsample(self, fmt: str, n: int, seed: int = 0, with_labels: bool = False, scikit_safe: bool = False) -> pathlib.Path:
+        """ Write subset of `n` samples from the test split to disk in `fmt` format
+
+        Iff `with_labels` is true, the target column will be included in the split file.
+        Iff `scikit_safe` is true, categorical values are encoded and missing values
+        are imputed.
+        """
+        # Just a hack for now, the splitters all work specifically with openml tasks.
+        # The important thing is that we split to disk and can load it later.
+
+        # We should consider taking a stratified sample if n is large enough,
+        # inference time might differ based on class
+        if scikit_safe:
+            if with_labels:
+                _, data = impute_array(self.train.data_enc, self.test.data_enc)
+            else:
+                _, data = impute_array(self.train.X_enc, self.test.X_enc)
+
+            columns = self._test.data.columns if with_labels else self._test.X.columns
+            data = pd.DataFrame(data, columns=columns)
+        else:
+            data = self._test.data if with_labels else self._test.X
+
+        subsample = data.sample(
+            n=n,
+            replace=True,
+            random_state=seed,
+        )
+
+        _, test_path = self._get_split_paths()
+        test_path = pathlib.Path(test_path)
+        subsample_path = test_path.parent / f"{test_path.stem}_{n}_{seed}.{fmt}"
+        if fmt == "csv":
+            subsample.to_csv(subsample_path, header=True, index=False)
+        elif fmt == "arff":
+            ArffSplitter(self)._save_split(
+                subsample,
+                subsample_path,
+                name=f"{self._oml_dataset.name}_inference_{self.fold}_{n}"
+            )
+        elif fmt == "parquet":
+            subsample.to_parquet(subsample_path)
+        else:
+            msg = f"{fmt=}, but must be one of 'csv', 'arff', or 'parquet'."
+            raise ValueError(msg)
+
+        return subsample_path
+
     @lazy_property
     @profile(logger=log)
     def features(self):
diff --git a/amlb/datautils.py b/amlb/datautils.py
index f6bd2ef90..7946fdc1c 100644
--- a/amlb/datautils.py
+++ b/amlb/datautils.py
@@ -26,7 +26,7 @@
 log = logging.getLogger(__name__)
 
 
-def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dtype=None):
+def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dtype=None, timestamp_column=None):
     """
     read csv file to DataFrame.
 
@@ -37,13 +37,19 @@ def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dty
     :param header: if the columns header should be read.
     :param as_data_frame: if the result should be returned as a data frame (default) or a numpy array.
     :param dtype: data type for columns.
+    :param timestamp_column: column name for timestamp, to ensure dates are correctly parsed by pandas.
     :return: a DataFrame
     """
+    if dtype is not None and timestamp_column is not None and timestamp_column in dtype:
+            dtype = dtype.copy() # to avoid outer context manipulation
+            del dtype[timestamp_column]
+
     df = pd.read_csv(path,
                      nrows=nrows,
                      header=0 if header else None,
                      index_col=0 if index else None,
-                     dtype=dtype)
+                     dtype=dtype,
+                     parse_dates=[timestamp_column] if timestamp_column is not None else None)
     return df if as_data_frame else df.values
 
 
@@ -171,6 +177,9 @@ def __init__(self, type='label', target=True, encoded_type=int,
         else:
             raise ValueError("Encoder `type` should be one of {}.".format(['label', 'one-hot']))
 
+    def __repr__(self):
+        return repr_def(self)
+
     @property
     def _ignore_missing(self):
         return self.for_target or self.missing_policy == 'ignore'
@@ -211,8 +220,8 @@ def transform(self, vec, **params):
         :param params:
         :return:
         """
-        if log.isEnabledFor(5):  # logging.TRACE
-            log.debug("Transforming %s using %s", vec, repr_def(self))
+        if log.isEnabledFor(logging.TRACE):
+            log.debug("Transforming %s using %s", vec, self)
 
         return_value = lambda v: v
         if isinstance(vec, str):
diff --git a/amlb/defaults.py b/amlb/defaults.py
new file mode 100644
index 000000000..6d0bf35c5
--- /dev/null
+++ b/amlb/defaults.py
@@ -0,0 +1,12 @@
+import pathlib
+
+from openml.config import cache_directory
+
+from amlb.utils import Namespace as ns
+
+default_dirs = ns(
+    input_dir=cache_directory,
+    output_dir=str(pathlib.Path(__file__).parent.parent / "results"),
+    user_dir="~/.config/automlbenchmark",
+    root_dir=str(pathlib.Path(__file__).parent.parent)
+)
diff --git a/amlb/frameworks/definitions.py b/amlb/frameworks/definitions.py
index c1f050a19..3bd3c59a4 100644
--- a/amlb/frameworks/definitions.py
+++ b/amlb/frameworks/definitions.py
@@ -22,7 +22,6 @@ def load_framework_definitions(frameworks_file: Union[str, List[str]], config: N
     frameworks = _load_and_merge_framework_definitions(frameworks_file, config)
     for tag, defs in frameworks:
         _sanitize_and_add_defaults(defs, config)
-    log.debug("Available framework definitions:\n%s", frameworks)
     return frameworks
 
 
@@ -102,6 +101,9 @@ def _add_default_setup_env(framework):
 
 
 def _add_default_setup_args(framework):
+    if "setup_args" in framework and isinstance(framework.setup_args, list):
+        return
+
     if "setup_args" in framework and isinstance(framework.setup_args, str):
         framework.setup_args = [framework.setup_args]
     else:
diff --git a/amlb/job.py b/amlb/job.py
index 01dd24f37..6701f033d 100644
--- a/amlb/job.py
+++ b/amlb/job.py
@@ -14,6 +14,8 @@
 import queue
 import signal
 import threading
+from functools import partial
+from typing import Callable, List, Optional
 
 from .utils import Namespace, Timer, ThreadSafeCounter, InterruptTimeout, is_main_thread, raise_in_thread, signal_handler
 
@@ -64,7 +66,10 @@ def is_state_transition_ok(cls, old_state: State, new_state: State):
 
     printer = pprint.PrettyPrinter(indent=2, compact=True)
 
-    def __init__(self, name="", timeout_secs=None, priority=None, raise_on_failure=False):
+    def __init__(self, name: str = "",
+                 timeout_secs: int = -1,
+                 priority: Optional[int] = None,
+                 raise_on_failure: bool = False):
         """
 
         :param name:
@@ -195,18 +200,20 @@ class JobRunner:
         (State.stopping,    [State.stopped]),
         (State.stopped,     None)
     ]
+    END_Q = object()
 
     @classmethod
     def is_state_transition_ok(cls, old_state: State, new_state: State):
         allowed = next((head for tail, head in cls.state_machine if tail == old_state), None)
         return allowed and new_state in allowed
 
-    def __init__(self, jobs):
+    def __init__(self, jobs: List, on_new_result: Optional[Callable] = None):
         self.jobs = jobs
         self.results = []
         self.state = None
         self._queue = None
         self._last_priority = 0
+        self._on_new_result = on_new_result
         self.set_state(State.created)
 
     def start(self):
@@ -235,7 +242,7 @@ def stop_if_complete(self):
         if 0 < len(self.jobs) == len(self.results):
             self.stop()
 
-    def put(self, job, priority=None):
+    def put(self, job: Job, priority: Optional[int] = None):
         if self.state in [State.stopping, State.stopped]:
             return
         if priority is None:
@@ -248,7 +255,7 @@ def put(self, job, priority=None):
         else:
             log.warning("Ignoring job `%s`. Runner state: `%s`", job.name, self.state)
 
-    def reschedule(self, job, priority=None):
+    def reschedule(self, job: Job, priority: Optional[int] = None):
         if self.state not in [State.running]:
             return
         job.reschedule()
@@ -267,6 +274,11 @@ def set_state(self, state: State):
             log.exception("Error when handling state change to %s for job runner: %s", state, str(e))
         return not skip_default
 
+    def _add_result(self, result):
+        self.results.append(result)
+        if self._on_new_result is not None:
+            self._on_new_result(result)
+
     def __iter__(self):
         return self
 
@@ -289,7 +301,7 @@ def _run(self):
 
     def _stop(self):
         if self._queue:
-            self._queue.put((-1, None))
+            self._queue.put((-1, JobRunner.END_Q))
         jobs = self.jobs.copy()
         self.jobs.clear()
         for job in jobs:
@@ -301,19 +313,19 @@ def _on_state(self, state: State):
 
 class SimpleJobRunner(JobRunner):
 
-    def __init__(self, jobs):
-        super().__init__(jobs)
+    def __init__(self, jobs: List, on_new_result: Optional[Callable] = None):
+        super().__init__(jobs, on_new_result=on_new_result)
         self._interrupt = threading.Event()
 
     def _run(self):
         for job in self:
-            if job is None or self._interrupt.is_set():
+            if job is JobRunner.END_Q or self._interrupt.is_set():
                 break
             result = job.start()
             if job.state is State.rescheduling:
                 self.reschedule(job)
             else:
-                self.results.append(result)
+                self._add_result(result)
                 job.done()
             self.stop_if_complete()
 
@@ -328,16 +340,43 @@ class QueueingStrategy:
         keep_queue_full = 0
         enforce_job_priority = 1
 
-    def __init__(self, jobs, parallel_jobs=1, done_async=True, delay_secs=0,
+    def __init__(self, jobs: List,
+                 on_new_result: Optional[Callable] = None,
+                 parallel_jobs: int = 1,
+                 done_async: bool = True,
+                 delay_secs: int = 0,
                  queueing_strategy: QueueingStrategy = QueueingStrategy.keep_queue_full,
-                 use_daemons=False):
-        super().__init__(jobs)
+                 use_daemons: bool = False):
+        super().__init__(jobs, on_new_result=on_new_result)
         self.parallel_jobs = parallel_jobs
         self._done_async = done_async
         self._delay = delay_secs  # short sleep between enqueued jobs to make console more readable
         self._daemons = use_daemons
         self._queueing_strategy = queueing_strategy
         self._interrupt = threading.Event()
+        self._exec = None
+        self.futures = []
+
+    def _safe_call_from_exec(self, fn):
+        if self._exec:
+            future = self._exec.submit(fn)
+            self.futures.append(future)
+        else:
+            log.warning(
+                "Application is submitting a function while the thread executor is not running: executing the function in the calling thread.")
+            try:
+                fn()
+            except Exception as e:
+                log.exception(e)
+
+    def _add_result(self, result):
+        sup_call = partial(super()._add_result, result)
+        self._safe_call_from_exec(sup_call)
+
+    def stop_if_complete(self):
+        # Direct calls introduce a race condition by the queued '_add_result' calls
+        sup_call = super().stop_if_complete
+        self._safe_call_from_exec(sup_call)
 
     def _run(self):
         q = queue.Queue()
@@ -349,13 +388,13 @@ def worker():
                 job = q.get()
                 available_workers.dec()
                 try:
-                    if job is None or self._interrupt.is_set():
+                    if job is JobRunner.END_Q or self._interrupt.is_set():
                         break
                     result = job.start()
                     if job.state is State.rescheduling:
                         self.reschedule(job)
                     else:
-                        self.results.append(result)
+                        self._add_result(result)
                         if self._done_async:
                             job.done()
                     self.stop_if_complete()
@@ -387,7 +426,7 @@ def worker():
             q.maxsize = self.parallel_jobs  # resize to ensure that all workers can get a None job
             for _ in range(self.parallel_jobs):
                 try:
-                    q.put_nowait(None)     # stopping workers
+                    q.put_nowait(JobRunner.END_Q)     # stopping workers
                 except:
                     pass
             for thread in threads:
@@ -397,8 +436,17 @@ def worker():
                     job.done()
 
     def _on_state(self, state: State):
+        if state is State.starting:
+            self._exec = ThreadPoolExecutor(max_workers=1, thread_name_prefix="job_runner_exec_")
         if state is State.stopping:
             self._interrupt.set()
+            if self._exec is not None:
+                try:
+                    self._exec.shutdown(wait=True)
+                except:
+                    pass
+                finally:
+                    self._exec = None
 
 
 class MultiProcessingJobRunner(JobRunner):
diff --git a/amlb/resources.py b/amlb/resources.py
index d910587bc..d8388c78c 100644
--- a/amlb/resources.py
+++ b/amlb/resources.py
@@ -132,11 +132,18 @@ def framework_definition(self, name, tag=None):
         if tag not in self._frameworks:
             raise ValueError("Incorrect tag `{}`: only those among {} are allowed.".format(tag, self.config.frameworks.tags))
         frameworks = self._frameworks[tag]
+        log.debug("Available framework definitions:\n%s", frameworks)
         framework = next((f for n, f in frameworks if n.lower() == lname), None)
+        # TODO: Clean up this workflow and error messaging as part of #518
+        base_framework = next((f for n, f in self._frameworks[default_tag] if n.lower() == lname), None)
+        if framework and framework['removed']:
+            raise ValueError(f"Framework definition `{name}` has been removed from the benchmark: {framework['removed']}")
+        if not framework and (base_framework and base_framework['removed']):
+            raise ValueError(f"Framework definition `{name}` has been removed from the benchmark: {base_framework['removed']}")
         if not framework:
-            raise ValueError("Incorrect framework `{}`: not listed in {}.".format(name, self.config.frameworks.definition_file))
+            raise ValueError(f"Incorrect framework `{name}`: not listed in {self.config.frameworks.definition_file}.")
         if framework['abstract']:
-            raise ValueError("Framework definition `{}` is abstract and cannot be run directly.".format(name))
+            raise ValueError(f"Framework definition `{name}` is abstract and cannot be run directly.")
         return framework, framework.name
 
     @lazy_property
@@ -217,7 +224,9 @@ def _validate_task(self, task, lenient=False):
                                  else None) or task.name) if task['dataset'] is not None
                           else None)
             if not lenient and task[conf] is None:
-                raise ValueError("task definition must contain one property among ['openml_task_id', 'dataset']")
+                raise ValueError("task definition must contain an ID or one property "
+                                 "among ['openml_task_id', 'dataset'] to create an ID, "
+                                 "but task definition is {task}".format(task=str(task)))
 
         conf = 'metric'
         if task[conf] is None:
diff --git a/amlb/results.py b/amlb/results.py
index 2f547b4ec..6e1a5bc60 100644
--- a/amlb/results.py
+++ b/amlb/results.py
@@ -55,8 +55,8 @@ class Scoreboard:
     results_file = 'results.csv'
 
     @classmethod
-    def all(cls, scores_dir=None):
-        return cls(scores_dir=scores_dir)
+    def all(cls, scores_dir=None, autoload=True):
+        return cls(scores_dir=scores_dir, autoload=autoload)
 
     @classmethod
     def from_file(cls, path):
@@ -105,28 +105,34 @@ def load_df(file):
     def save_df(data_frame, path, append=False):
         exists = os.path.isfile(path)
         new_format = False
+        df = data_frame
         if exists:
-            df = read_csv(path, nrows=1)
-            new_format = list(df.columns) != list(data_frame.columns)
+            head = read_csv(path, nrows=1)
+            new_format = list(head.columns) != list(data_frame.columns)
         if new_format or (exists and not append):
             backup_file(path)
+        if new_format and append:
+            df = read_csv(path).append(data_frame)
         new_file = not exists or not append or new_format
         is_default_index = data_frame.index.name is None and not any(data_frame.index.names)
         log.debug("Saving scores to `%s`.", path)
-        write_csv(data_frame,
+        write_csv(df,
                   path=path,
                   header=new_file,
                   index=not is_default_index,
                   append=not new_file)
         log.info("Scores saved to `%s`.", path)
 
-    def __init__(self, scores=None, framework_name=None, benchmark_name=None, task_name=None, scores_dir=None):
+    def __init__(self, scores=None, framework_name=None, benchmark_name=None, task_name=None,
+                 scores_dir=None, autoload=True):
         self.framework_name = framework_name
         self.benchmark_name = benchmark_name
         self.task_name = task_name
         self.scores_dir = (scores_dir if scores_dir
                            else output_dirs(rconfig().output_dir, rconfig().sid, ['scores']).scores)
-        self.scores = scores if scores is not None else self._load()
+        self.scores = (scores if scores is not None
+                       else self.load_df(self.path) if autoload
+                       else None)
 
     @cached
     def as_data_frame(self):
@@ -155,24 +161,27 @@ def as_data_frame(self):
 
     @memoize
     def as_printable_data_frame(self, verbosity=3):
-        str_print = lambda val: '' if val in [None, '', 'None'] or (isinstance(val, float) and np.isnan(val)) else val
-        int_print = lambda val: int(val) if isinstance(val, float) and not np.isnan(val) else str_print(val)
-        num_print = lambda fn, val: None if isinstance(val, str) else fn(val)
+        str_print = lambda val: '' if val in [None, '', 'None'] or (isinstance(val, float) and np.isnan(val)) else str(val)
+        int_print = lambda val: int(val) if isinstance(val, (float, int)) and not np.isnan(val) else str_print(val)
 
         df = self.as_data_frame()
+        if df.empty:
+            return df
+
         force_str_cols = ['id']
         nanable_int_cols = ['fold', 'models_count', 'seed']
         low_precision_float_cols = ['duration', 'training_duration', 'predict_duration']
-        high_precision_float_cols = [col for col in df.select_dtypes(include=[np.float]).columns if col not in ([] + nanable_int_cols + low_precision_float_cols)]
+        high_precision_float_cols = [col for col in df.select_dtypes(include=[float]).columns if col not in ([] + nanable_int_cols + low_precision_float_cols)]
         for col in force_str_cols:
-            df[col] = df[col].astype(np.object).map(str_print).astype(np.str)
+            df[col] = df[col].map(str_print)
         for col in nanable_int_cols:
-            df[col] = df[col].astype(np.object).map(int_print).astype(np.str)
+            df[col] = df[col].map(int_print)
         for col in low_precision_float_cols:
             float_format = lambda f: ("{:.1g}" if f < 1 else "{:.1f}").format(f)
-            df[col] = df[col].astype(np.float).map(partial(num_print, float_format)).astype(np.float)
+            # The .astype(float) is required to maintain NaN as 'NaN' instead of 'nan'
+            df[col] = df[col].map(float_format).astype(float)
         for col in high_precision_float_cols:
-            df[col] = df[col].map(partial(num_print, "{:.6g}".format)).astype(np.float)
+            df[col] = df[col].map("{:.6g}".format).astype(float)
 
         cols = ([] if verbosity == 0
                 else ['task', 'fold', 'framework', 'constraint', 'result', 'metric', 'info'] if verbosity == 1
@@ -181,15 +190,17 @@ def as_printable_data_frame(self, verbosity=3):
                 else slice(None))
         return df.loc[:, cols]
 
-    def _load(self):
-        return self.load_df(self._score_file())
+    def load(self):
+        self.scores = self.load_df(self.path)
+        return self
 
     def save(self, append=False):
-        self.save_df(self.as_printable_data_frame(), path=self._score_file(), append=append)
+        self.save_df(self.as_printable_data_frame(), path=self.path, append=append)
+        return self
 
     def append(self, board_or_df, no_duplicates=True):
         to_append = board_or_df.as_data_frame() if isinstance(board_or_df, Scoreboard) else board_or_df
-        scores = self.as_data_frame().append(to_append, sort=False)
+        scores = self.as_data_frame().append(to_append)
         if no_duplicates:
             scores = scores.drop_duplicates()
         return Scoreboard(scores=scores,
@@ -198,7 +209,8 @@ def append(self, board_or_df, no_duplicates=True):
                           task_name=self.task_name,
                           scores_dir=self.scores_dir)
 
-    def _score_file(self):
+    @property
+    def path(self):
         sep = rconfig().token_separator
         if self.framework_name:
             if self.task_name:
@@ -228,12 +240,17 @@ def load_predictions(predictions_file):
             try:
                 df = read_csv(predictions_file, dtype=object)
                 log.debug("Predictions preview:\n %s\n", df.head(10).to_string())
+
                 if rconfig().test_mode:
                     TaskResult.validate_predictions(df)
-                if df.shape[1] > 2:
-                    return ClassificationResult(df)
+
+                if  'y_past_period_error' in df.columns:
+                    return TimeSeriesResult(df)
                 else:
-                    return RegressionResult(df)
+                    if df.shape[1] > 2:
+                        return ClassificationResult(df)
+                    else:
+                        return RegressionResult(df)
             except Exception as e:
                 return ErrorResult(ResultError(e))
         else:
@@ -254,6 +271,7 @@ def load_metadata(metadata_file):
     def save_predictions(dataset: Dataset, output_file: str,
                          predictions: Union[A, DF, S] = None, truth: Union[A, DF, S] = None,
                          probabilities: Union[A, DF] = None, probabilities_labels: Union[list, A] = None,
+                         optional_columns: Union[A, DF] = None,
                          target_is_encoded: bool = False,
                          preview: bool = True):
         """ Save class probabilities and predicted labels to file in csv format.
@@ -264,6 +282,7 @@ def save_predictions(dataset: Dataset, output_file: str,
         :param predictions:
         :param truth:
         :param probabilities_labels:
+        :param optional_columns:
         :param target_is_encoded:
         :param preview:
         :return: None
@@ -308,6 +327,10 @@ def save_predictions(dataset: Dataset, output_file: str,
 
         df = df.assign(predictions=preds)
         df = df.assign(truth=truth)
+
+        if optional_columns is not None:
+            df = pd.concat([df, optional_columns], axis=1)
+
         if preview:
             log.info("Predictions preview:\n %s\n", df.head(20).to_string())
         backup_file(output_file)
@@ -421,6 +444,11 @@ def compute_score(self, result=None, meta_result=None):
         required_meta_res = ['training_duration', 'predict_duration', 'models_count']
         for m in required_meta_res:
             entry[m] = meta_result[m] if m in meta_result else nan
+
+        if inference_times := Namespace.get(meta_result, "inference_times"):
+            for data_type, measurements in Namespace.dict(inference_times).items():
+                for n_samples, measured_times in Namespace.dict(measurements).items():
+                    entry[f"infer_batch_size_{data_type}_{n_samples}"] = np.median(measured_times)
         result = self.get_result() if result is None else result
 
         scoring_errors = []
@@ -450,7 +478,7 @@ def set_score(score):
         entry.info = result.info
         if scoring_errors:
             entry.info = "; ".join(filter(lambda it: it, [entry.info, *scoring_errors]))
-        entry |= Namespace({k: v for k, v in meta_result if k not in required_meta_res})
+        entry |= Namespace({k: v for k, v in meta_result if k not in required_meta_res and k != "inference_times"})
         log.info("Metric scores: %s", entry)
         return entry
 
@@ -656,6 +684,71 @@ def r2(self):
         """R^2"""
         return float(r2_score(self.truth, self.predictions))
 
+class TimeSeriesResult(RegressionResult):
+
+    def __init__(self, predictions_df, info=None):
+        super().__init__(predictions_df, info)
+        self.truth = self.df['truth'].values if self.df is not None else None #.iloc[:, 1].values if self.df is not None else None
+        self.predictions = self.df['predictions'].values if self.df is not None else None #.iloc[:, -2].values if self.df is not None else None
+        self.y_past_period_error = self.df['y_past_period_error'].values
+        self.quantiles = self.df.iloc[:, 2:-1].values
+        self.quantiles_probs = np.array([float(q) for q in self.df.columns[2:-1]])
+        self.truth = self.truth.astype(float, copy=False)
+        self.predictions = self.predictions.astype(float, copy=False)
+        self.quantiles = self.quantiles.astype(float, copy=False)
+        self.y_past_period_error = self.y_past_period_error.astype(float, copy=False)
+
+        self.target = Feature(0, 'target', 'real', is_target=True)
+        self.type = DatasetType.timeseries
+
+    @metric(higher_is_better=False)
+    def mase(self):
+        """Mean Absolute Scaled Error"""
+        return float(np.nanmean(np.abs(self.truth/self.y_past_period_error - self.predictions/self.y_past_period_error)))
+
+    @metric(higher_is_better=False)
+    def smape(self):
+        """Symmetric Mean Absolute Percentage Error"""
+        num = np.abs(self.truth - self.predictions)
+        denom = (np.abs(self.truth) + np.abs(self.predictions)) / 2
+        # If the denominator is 0, we set it to float('inf') such that any division yields 0 (this
+        # might not be fully mathematically correct, but at least we don't get NaNs)
+        denom[denom == 0] = math.inf
+        return np.mean(num / denom)
+
+    @metric(higher_is_better=False)
+    def mape(self):
+        """Symmetric Mean Absolute Percentage Error"""
+        num = np.abs(self.truth - self.predictions)
+        denom = np.abs(self.truth)
+        # If the denominator is 0, we set it to float('inf') such that any division yields 0 (this
+        # might not be fully mathematically correct, but at least we don't get NaNs)
+        denom[denom == 0] = math.inf
+        return np.mean(num / denom)
+
+    @metric(higher_is_better=False)
+    def nrmse(self):
+        """Normalized Root Mean Square Error"""
+        return self.rmse() / np.mean(np.abs(self.truth))
+
+    @metric(higher_is_better=False)
+    def wape(self):
+        """Weighted Average Percentage Error"""
+        return np.sum(np.abs(self.truth - self.predictions)) / np.sum(np.abs(self.truth))
+
+    @metric(higher_is_better=False)
+    def ncrps(self):
+        """Normalized Continuous Ranked Probability Score"""
+        quantile_losses = 2 * np.sum(
+            np.abs(
+                (self.quantiles - self.truth[:, None])
+                * ((self.quantiles >= self.truth[:, None]) - self.quantiles_probs[None, :])
+            ),
+            axis=0,
+        )
+        denom = np.sum(np.abs(self.truth)) # shape [num_time_series, num_quantiles]
+        weighted_losses = quantile_losses.sum(0) / denom  # shape [num_quantiles]
+        return weighted_losses.mean()
 
 _encode_predictions_and_truth_ = False
 
diff --git a/amlb/runners/aws.py b/amlb/runners/aws.py
index 4368efde3..fec734a8a 100644
--- a/amlb/runners/aws.py
+++ b/amlb/runners/aws.py
@@ -196,7 +196,7 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
                 try:
                     jobs = flatten([self._make_aws_job([task_def.name], folds) for task_def in task_defs])
                     results = self._run_jobs(jobs)
-                    return self._process_results(results, task_name=tasks)
+                    return self._results_summary(self._process_results(results))
                 finally:
                     self.cleanup()
             else:
@@ -208,7 +208,8 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
                 task_names = None if tasks is None else [task_def.name for task_def in task_defs]
                 job = self._make_aws_job(task_names, folds)
                 results = self._run_jobs([job])
-                return self._process_results(results, task_name=tasks)
+                scoreboard = self._process_results(results)
+                return self._results_summary(scoreboard)
             finally:
                 self.cleanup()
 
@@ -217,7 +218,8 @@ def _create_job_runner(self, jobs):
             return SimpleJobRunner(jobs)
         else:
             queueing_strategy = MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority
-            return MultiThreadingJobRunner(jobs, self.parallel_jobs,
+            return MultiThreadingJobRunner(jobs,
+                                           parallel_jobs=self.parallel_jobs,
                                            delay_secs=rconfig().job_scheduler.delay_between_jobs,
                                            done_async=True,
                                            queueing_strategy=queueing_strategy)
@@ -574,6 +576,8 @@ def _start_instance(self, instance_def, script_params="", instance_key=None, tim
             if instance_def.volume_size:
                 ebs['VolumeSize'] = instance_def.volume_size
 
+            instance_tags = ec2_config.instance_tags | ns(Name=f"amlb_{inst_key}")
+            volume_tags = (ec2_config.volume_tags or instance_tags) | ns(Name=f"amlb_{inst_key}")
             instance_params = dict(
                 BlockDeviceMappings=[dict(
                     DeviceName=ec2_config.root_device_name,
@@ -588,15 +592,11 @@ def _start_instance(self, instance_def, script_params="", instance_key=None, tim
                 TagSpecifications=[
                     dict(
                         ResourceType='instance',
-                        Tags=[
-                            dict(Key='Name', Value=f"benchmark_{inst_key}")
-                        ]
+                        Tags=[dict(Key=k, Value=v) for k, v in instance_tags]
                     ),
                     dict(
                         ResourceType='volume',
-                        Tags=[
-                            dict(Key='Name', Value=f"benchmark_{inst_key}")
-                        ]
+                        Tags=[dict(Key=k, Value=v) for k, v in volume_tags]
                     ),
                 ],
                 UserData=self._ec2_startup_script(inst_key, script_params=script_params, timeout_secs=timeout_secs)
@@ -902,10 +902,8 @@ def download_file(obj, dest, dest_display_path=None):
                 dest_path = os.path.join(self.output_dirs.session, rel_path)
                 try:
                     download_file(obj, dest_path)
-                    # if obj.key == result_key:
                     if is_result and not success:
-                        if rconfig().results.save:
-                            self._exec_send(lambda path: self._append(Scoreboard.load_df(path)), dest_path)
+                        self._exec_send(lambda path: self._save_global(Scoreboard.from_file(path)), dest_path)
                         success = True
                 except Exception as e:
                     if is_result:
@@ -926,6 +924,11 @@ def download_file(obj, dest, dest_display_path=None):
         self._update_instance(instance_id, success=success)
         return success, error
 
+    def _results_summary(self, scoreboard=None):
+        log.info(
+            "Result summary not available for AWS mode (reference files instead)."
+        )
+
     def _create_instance_profile(self):
         """
         see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
@@ -1107,7 +1110,7 @@ def _ec2_startup_script(self, instance_key, script_params="", timeout_secs=-1):
   - pip3 install -U awscli wheel
   - aws s3 cp '{s3_input}' /s3bucket/input --recursive
   - aws s3 cp '{s3_user}' /s3bucket/user --recursive
-  - docker run {docker_options} -v /s3bucket/input:/input -v /s3bucket/output:/output -v /s3bucket/user:/custom --rm {image} {params} -i /input -o /output -u /custom -s skip -Xrun_mode=aws.docker {extra_params}
+  - docker run {docker_options} -v /s3bucket/input:/input -v /s3bucket/output:/output -u "0:0" -v /s3bucket/user:/custom --rm {image} {params} -i /input -o /output -u /custom -s skip -Xrun_mode=aws.docker {extra_params}
   - aws s3 cp /s3bucket/output '{s3_output}' --recursive
   #- rm -f /var/lib/cloud/instance/sem/config_scripts_user
 
diff --git a/amlb/runners/container.py b/amlb/runners/container.py
index f33f220ab..c4a6110e0 100644
--- a/amlb/runners/container.py
+++ b/amlb/runners/container.py
@@ -93,7 +93,8 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
             job = self._make_container_job(tasks, folds)
             try:
                 results = self._run_jobs([job])
-                return self._process_results(results, task_name=tasks)
+                scoreboard = self._process_results(results)
+                return self._results_summary(scoreboard)
             finally:
                 self.cleanup()
 
diff --git a/amlb/runners/docker.py b/amlb/runners/docker.py
index 2aef93bce..491c7c7cb 100644
--- a/amlb/runners/docker.py
+++ b/amlb/runners/docker.py
@@ -9,7 +9,7 @@
 import re
 
 from ..benchmark import _setup_dir_
-from ..resources import config as rconfig, get as rget
+from ..resources import config as rconfig
 from ..utils import dir_of, run_cmd, str_digest, str_sanitize, touch
 from .container import ContainerBenchmark
 
@@ -31,6 +31,12 @@ def __init__(self, framework_name, benchmark_name, constraint_name):
         """
         super().__init__(framework_name, benchmark_name, constraint_name)
         self._custom_image_name = rconfig().docker.image
+        self.user = os.getlogin()
+        # For linux specifically, files created within docker are not
+        # automatically owned by the user starting the docker instance.
+        # For Windows permissions are set fine, so we don't need user information.
+        self.userid = None if os.name == 'nt' else os.getuid()
+        self.usergid =  None if os.name == 'nt' else os.getgid()
         self.minimize_instances = rconfig().docker.minimize_instances
         self.container_name = 'docker'
         self.force_branch = rconfig().docker.force_branch
@@ -53,12 +59,13 @@ def _start_container(self, script_params=""):
         script_extra_params = "--session="  # in combination with `self.output_dirs.session` usage below to prevent creation of 2 sessions locally
         inst_name = f"{self.sid}.{str_sanitize(str_digest(script_params))}"
         cmd = (
-            "docker run --name {name} {options} "
+            "docker run --name {name} {options} {run_as_user}"
             "-v {input}:/input -v {output}:/output -v {custom}:/custom "
             "--rm {image} {params} -i /input -o /output -u /custom -s skip -Xrun_mode=docker {extra_params}"
         ).format(
             name=inst_name,
             options=rconfig().docker.run_extra_options,
+            run_as_user='' if os.name == 'nt' else f'-u "{self.userid}:{self.usergid}" ',
             input=in_dir,
             output=self.output_dirs.session,
             custom=custom_dir,
@@ -94,12 +101,14 @@ def _image_exists(self, image):
 
     def _run_container_build_command(self, image, cache):
         log.info(f"Building docker image {image}.")
-        run_cmd("docker build {options} -t {container} -f {script} .".format(
+        run_cmd("docker build {options} {build_extra_options} -t {container} -f {script} .".format(
             options="" if cache else "--no-cache",
             container=image,
-            script=self._script),
+            script=self._script,
+            build_extra_options=rconfig().docker.build_extra_options,
+        ),
             _live_output_=rconfig().setup.live_output,
-            _activity_timeout_=rconfig().setup.activity_timeout
+            _activity_timeout_=rconfig().setup.activity_timeout,
         )
         log.info(f"Successfully built docker image {image}.")
 
@@ -109,18 +118,23 @@ def _upload_image(self, image):
         log.info(f"Successfully published docker image {image}.")
 
     def _generate_script(self, custom_commands):
-        docker_content = """FROM ubuntu:18.04
+        docker_content = """FROM ubuntu:22.04
 
 ENV DEBIAN_FRONTEND noninteractive
 RUN apt-get update
-RUN apt-get -y install apt-utils dialog locales
+RUN apt-get -y install apt-utils dialog locales tzdata sudo
 RUN apt-get -y install curl wget unzip git
 RUN apt-get -y install software-properties-common
 RUN add-apt-repository -y ppa:deadsnakes/ppa
 RUN apt-get update
 RUN apt-get -y install python{pyv} python{pyv}-venv python{pyv}-dev python3-pip
+RUN apt-get -y install libhdf5-serial-dev
 #RUN update-alternatives --install /usr/bin/python3 python3 $(which python{pyv}) 1
 
+RUN adduser --disabled-password --gecos '' {userid_option} {username}
+RUN adduser {username} sudo
+RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
+
 # aliases for the python system
 ENV SPIP python{pyv} -m pip
 ENV SPY python{pyv}
@@ -132,6 +146,9 @@ def _generate_script(self, custom_commands):
 ENV LANG C.UTF-8
 ENV LC_ALL C.UTF-8
 
+# Setup HDF5 for installing `tables`
+ENV HDF5_DIR /usr/lib/aarch64-linux-gnu/hdf5/serial
+
 WORKDIR /bench
 
 # We create a virtual environment so that AutoML systems may use their preferred versions of
@@ -151,6 +168,9 @@ def _generate_script(self, custom_commands):
 ADD . /bench/
 
 RUN (grep -v '^\\s*#' | xargs -L 1 $PIP install --no-cache-dir) < requirements.txt
+RUN chown -R {username}:{username} /bench
+RUN chown -R {username}:{username} /home/{username}
+USER {username}
 
 RUN $PY {script} {framework} -s only
 {custom_commands}
@@ -170,7 +190,8 @@ def _generate_script(self, custom_commands):
             pyv=rconfig().versions.python,
             pipv=rconfig().versions.pip,
             script=rconfig().script,
-            user=rconfig().user_dir,
+            userid_option=f"-uid {self.userid}" if self.userid else '',
+            username=self.user,
         )
 
         touch(self._script)
diff --git a/amlb/runners/singularity.py b/amlb/runners/singularity.py
index 672db3770..451fccadb 100644
--- a/amlb/runners/singularity.py
+++ b/amlb/runners/singularity.py
@@ -155,7 +155,7 @@ def _upload_image(self, image):
 
     def _generate_script(self, custom_commands):
         singularity_content = """Bootstrap: docker
-From: ubuntu:18.04
+From: ubuntu:22.04
 %files
 . /bench/
 %post
diff --git a/amlb/uploads.py b/amlb/uploads.py
new file mode 100644
index 000000000..693500054
--- /dev/null
+++ b/amlb/uploads.py
@@ -0,0 +1,191 @@
+import json
+import logging
+import pathlib
+from collections import OrderedDict
+import textwrap
+from typing import Set, Optional, List
+
+import openml
+import pandas as pd
+from openml import OpenMLTask, OpenMLFlow
+from openml.runs.functions import format_prediction
+
+from .utils.core import Namespace
+
+log = logging.getLogger(__name__)
+
+
+def _load_task_data(task_folder: pathlib.Path, fold: int = 0) -> Namespace:
+    """ Loads the metadata of the given fold of a task as a namespace. """
+    with open(task_folder / str(fold) / 'metadata.json', 'r') as fh:
+        metadata = json.load(fh)
+    metadata = Namespace.from_dict(metadata)
+    return metadata
+
+
+def _load_fold(task_folder: pathlib.Path, fold: int, task: OpenMLTask) -> pd.DataFrame:
+    """ Load the predictions and add openml repeat/fold/index information. """
+    prediction_file = task_folder / f"{fold}/predictions.csv"
+    predictions = pd.read_csv(prediction_file, sep=',', header=0)
+
+    train_indices, test_indices = task.get_train_test_split_indices(fold, repeat=0, sample=0)
+    predictions["index"] = test_indices
+    predictions["fold"] = fold
+    predictions["repeat"] = 0
+    return predictions
+
+
+def _load_predictions(task_folder: pathlib.Path) -> pd.DataFrame:
+    """ Loads predictions of all folds for a task with index information required for upload. """
+    metadata = _load_task_data(task_folder)
+    task = openml.tasks.get_task(metadata.openml_task_id)
+    results = [_load_fold(task_folder, fold, task) for fold in range(10)]
+    return pd.concat(results)
+
+
+def _list_completed_folds(task_folder: pathlib.Path) -> Set[str]:
+    completed_folds = set()
+    for fold_dir in [d for d in task_folder.iterdir() if d.is_dir()]:
+        if (task_folder / fold_dir / "predictions.csv").is_file():
+            completed_folds.add(fold_dir.name)
+    return completed_folds
+
+
+def _get_flow(metadata: Namespace, sync_with_server: bool = True) -> openml.flows.OpenMLFlow:
+    """ Creates or retrieves an OpenML flow for the given run metadata. """
+    if metadata.git_info.repo == "NA":
+        msg = "Can't generate flow for results since repository information is unknown."
+        raise ValueError(msg)
+
+    amlb_version = next(iter(metadata.git_info.tags), metadata.git_info.commit[:8])
+
+    *_, owner, _ = metadata.git_info.repo.rsplit('/')
+    if owner != 'openml':
+        amlb_version = f"{owner}#{amlb_version}"
+
+    flow_description = """
+        This flow is generated by {fork}the automl benchmark: {repo}
+        Repository commit: {commit}
+        {framework_name} version: {framework_version}
+    """.format(
+        fork="a fork of " if owner != 'openml' else '',
+        repo=metadata.git_info.repo,
+        commit=metadata.git_info.commit,
+        framework_name=metadata.framework,
+        framework_version=metadata.framework_version,
+    )
+
+    amlb_flow = openml.flows.OpenMLFlow(
+        name=f"amlb_{metadata.framework}",
+        description=textwrap.dedent(flow_description),
+        external_version=f'amlb=={amlb_version},{metadata.framework}=={metadata.framework_version}',
+        # The values below are default values for a flow, the run will record used values.
+        parameters=OrderedDict(
+            max_runtime_seconds='14400',
+            max_mem_size_mb='32768',
+            cores='8',
+            seed='42',
+        ),
+        parameters_meta_info=OrderedDict(
+            max_runtime_seconds=dict(data_type='int', description='Maximum runtime in seconds.'),
+            max_mem_size_mb=dict(data_type='int', description='Memory constraint in megabytes.'),
+            cores=dict(data_type='int', description='Number of available cores.'),
+            seed=dict(data_type='int', description='The random seed.')
+        ),
+        language='English',
+        # We can use components to describe subflows, e.g. the automl framework with its hyperparameters.
+        # For now we don't.
+        components=OrderedDict(),
+        model=None,
+        tags=["amlb"],
+        dependencies=f'amlb=={amlb_version},{metadata.framework}=={metadata.framework_version}',
+    )
+    if sync_with_server:
+        # Publish will check if the flow exists on the server.
+        # If it exists, the local object is updated with server information.
+        # Otherwise the new flow is stored on the server.
+        return amlb_flow.publish()
+    else:
+        return amlb_flow
+
+
+def _extract_and_format_hyperparameter_configuration(metadata: Namespace, flow: OpenMLFlow) -> List[OrderedDict]:
+    return [
+        OrderedDict([
+            ("oml:name", name),
+            ("oml:value", metadata.__dict__.get(name, default)),
+            ("oml:component", flow.id)
+        ])
+        for name, default in flow.parameters.items()
+    ]
+
+
+def _upload_results(task_folder: pathlib.Path) -> openml.runs.OpenMLRun:
+    metadata = _load_task_data(task_folder)
+    predictions = _load_predictions(task_folder)
+
+    oml_flow = _get_flow(metadata)
+    oml_task = openml.tasks.get_task(metadata.openml_task_id)
+
+    denormalize_map = {label.strip().lower(): label for label in oml_task.class_labels}
+    predictions.columns = [col if col not in denormalize_map else denormalize_map[col] for col in predictions]
+    predictions["truth"] = predictions["truth"].apply(denormalize_map.get)
+    predictions["predictions"] = predictions["predictions"].apply(denormalize_map.get)
+
+    formatted_predictions = []
+    for _, row in predictions.iterrows():
+        if metadata.type != "classification":
+            class_probabilities = None
+        else:
+            class_probabilities = {c: row[c] for c in oml_task.class_labels}
+        prediction = format_prediction(
+               task=oml_task,
+               repeat=row["repeat"],
+               fold=row["fold"],
+               index=row["index"],
+               prediction=row["predictions"],
+               truth=row["truth"],
+               proba=class_probabilities,
+           )
+        formatted_predictions.append(prediction)
+
+    parameters = _extract_and_format_hyperparameter_configuration(metadata, oml_flow)
+    tags = ['amlb']
+    if 'tag' in metadata and metadata.tag not in [None, 'amlb']:
+        tags.extend([metadata.tag])
+
+    if len(metadata.framework_params) > 0:
+        description = "Framework_Hyperparameters-"
+        description += '-'.join(f"{hp}:{value}" for hp, value in Namespace.dict(metadata.framework_params).items())
+    else:
+        description = None
+
+    return openml.runs.OpenMLRun(
+        task_id=oml_task.id,
+        flow_id=oml_flow.id,
+        dataset_id=oml_task.dataset_id,
+        parameter_settings=parameters,
+        setup_string=metadata.command,
+        data_content=formatted_predictions,
+        tags=tags,
+        run_details=description,
+    ).publish()
+
+
+def missing_folds(task_folder: pathlib.Path) -> Set[str]:
+    completed_folds = _list_completed_folds(task_folder)
+    all_folds = {str(f) for f in range(10)}
+    return all_folds.difference(completed_folds)
+
+
+def process_task_folder(task_folder: pathlib.Path) -> Optional[openml.runs.OpenMLRun]:
+    """ Uploads the task folder iff predictions for all 10 folds are present. """
+    if len(missing_folds(task_folder)) > 0:
+        log.warning(
+            "Task %s is missing predictions for folds %s.",
+            task_folder,
+            ', '.join(missing_folds(task_folder))
+        )
+        return None
+
+    return _upload_results(task_folder)
diff --git a/amlb/utils/cache.py b/amlb/utils/cache.py
index 8c4282c69..00bc612a9 100644
--- a/amlb/utils/cache.py
+++ b/amlb/utils/cache.py
@@ -4,6 +4,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 _CACHE_PROP_PREFIX_ = '__cached__'
 
 
@@ -77,3 +79,4 @@ def decorator(self):
     return decorator
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/config.py b/amlb/utils/config.py
index 97b26ebc7..59bf7db30 100644
--- a/amlb/utils/config.py
+++ b/amlb/utils/config.py
@@ -1,42 +1,49 @@
 from collections import namedtuple
 from copy import deepcopy
 from dataclasses import dataclass
+from importlib.util import find_spec
 import logging
 import os
 from typing import Callable, List, Union
 
-from ruamel.yaml.constructor import SafeConstructor
-from ruamel.yaml.main import YAML
-
 from .core import Namespace, identity, json_load
 from .os import normalize_path
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+if find_spec('ruamel') is not None:
+    from ruamel.yaml.constructor import SafeConstructor
+    from ruamel.yaml.main import YAML
+    __no_export |= set(dir())
 
-class YAMLNamespaceConstructor(SafeConstructor):
+    class _YAMLNamespaceConstructor(SafeConstructor):
 
-    @classmethod
-    def init(cls):
-        cls.add_constructor(u'tag:yaml.org,2002:map', cls.construct_yaml_map)
+        @classmethod
+        def init(cls):
+            cls.add_constructor(u'tag:yaml.org,2002:map', cls.construct_yaml_map)
 
-    def construct_yaml_map(self, node):
-        data = Namespace()
-        yield data
-        value = self.construct_mapping(node)
-        data += value
+        def construct_yaml_map(self, node):
+            data = Namespace()
+            yield data
+            value = self.construct_mapping(node)
+            data += value
 
 
-YAMLNamespaceConstructor.init()
+    _YAMLNamespaceConstructor.init()
 
 
-def yaml_load(file, as_namespace=False):
-    if as_namespace:
-        yaml = YAML(typ='safe', pure=True)
-        yaml.Constructor = YAMLNamespaceConstructor
-    else:
-        yaml = YAML(typ='safe')
-    return yaml.load(file)
+    def yaml_load(file, as_namespace=False):
+        if as_namespace:
+            yaml = YAML(typ='safe', pure=True)
+            yaml.Constructor = _YAMLNamespaceConstructor
+        else:
+            yaml = YAML(typ='safe')
+        return yaml.load(file)
+else:
+    def yaml_load(*_, **__):
+        raise ImportError("ruamel.yaml package is required to load `yaml` config files.")
 
 
 def config_load(path, verbose=False):
@@ -87,3 +94,5 @@ def transform_config(config: Namespace, transform_rules: [TransformRule], inplac
                 Namespace.delete(config, k)
     return config
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/core.py b/amlb/utils/core.py
index 55469c9f6..e37f25984 100644
--- a/amlb/utils/core.py
+++ b/amlb/utils/core.py
@@ -11,22 +11,10 @@
 import re
 import sys
 import threading
-import types
 
 log = logging.getLogger(__name__)
 
-
-def register_module(module_name):
-    if module_name not in sys.modules:
-        mod = types.ModuleType(module_name)
-        sys.modules[module_name] = mod
-    return sys.modules[module_name]
-
-
-def register_submodule(mod, name):
-    fullname = '.'.join([mod.__name__, name])
-    module = register_module(fullname)
-    setattr(mod, name, module)
+__no_export = set(dir())  # all variables defined above this are not exported
 
 
 class Namespace:
@@ -234,11 +222,29 @@ def __json__(self):
         return Namespace.dict(self)
 
 
-def repr_def(obj, show_private=False):
+def _attributes(obj, filtr='all'):
+    attrs = vars(obj)
+    if filtr is None or filtr == 'all':
+        return attrs
+    elif filtr == 'public':
+        return {k: v for k, v in attrs.items() if not k.startswith('_')}
+    elif filtr == 'private':
+        return {k: v for k, v in attrs.items() if k.startswith('_')}
+    elif isinstance(filtr, list):
+        return {k: v for k, v in attrs.items() if k in filtr}
+    else:
+        assert callable(filtr)
+        return {k: v for k, v in attrs.items() if filtr(k)}
+
+
+def _classname(obj):
+    return type(obj).__name__
+
+
+def repr_def(obj, attributes='public'):
     return "{cls}({attrs!r})".format(
-        cls=type(obj).__name__,
-        attrs=(vars(obj) if show_private
-               else {k: v for k, v in vars(obj).items() if k.startswith('_')})
+        cls=_classname(obj),
+        attrs=_attributes(obj, attributes)
     )
 
 
@@ -460,3 +466,5 @@ def gen(*args, **kwargs):
         return threadsafe_iterator(fn(*args, **kwargs))
     return gen
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/modules.py b/amlb/utils/modules.py
index 0010409ff..4a2a1c4a6 100644
--- a/amlb/utils/modules.py
+++ b/amlb/utils/modules.py
@@ -1,4 +1,6 @@
 import logging
+import sys
+import types
 
 try:
     from pip._internal import main as pip_main
@@ -8,6 +10,21 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+
+def register_module(module_name):
+    if module_name not in sys.modules:
+        mod = types.ModuleType(module_name)
+        sys.modules[module_name] = mod
+    return sys.modules[module_name]
+
+
+def register_submodule(mod, name):
+    fullname = '.'.join([mod.__name__, name])
+    module = register_module(fullname)
+    setattr(mod, name, module)
+
 
 def pip_install(module_or_requirements, is_requirements=False):
     try:
@@ -19,3 +36,5 @@ def pip_install(module_or_requirements, is_requirements=False):
         log.error("Error when trying to install python modules %s.", module_or_requirements)
         log.exception(se)
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/os.py b/amlb/utils/os.py
index 39bd33f35..08c686a0b 100644
--- a/amlb/utils/os.py
+++ b/amlb/utils/os.py
@@ -12,6 +12,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def to_mb(size_in_bytes):
     return size_in_bytes / (1 << 20)
@@ -174,3 +176,4 @@ def __exit__(self, *args):
         # pass
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/process.py b/amlb/utils/process.py
index 593ed1ff8..d50d8615d 100644
--- a/amlb/utils/process.py
+++ b/amlb/utils/process.py
@@ -23,11 +23,30 @@
 import psutil
 
 from .core import Namespace, as_list, flatten, fn_name
-from .os import dir_of, to_mb
+from .os import dir_of, to_mb, path_from_split, split_path
 from .time import Timeout, Timer
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+
+@contextmanager
+def file_lock(path, timeout=-1):
+    """
+    :param path: the path of the file to lock.
+        A matching lock file is automatically generated and associated to the file that we want to manipulate.
+    :param timeout: timeout in seconds to wait for the lock to be acquired. Disabled by default.
+    :raise: Timeout if the lock could not be acquired after timeout.
+    """
+    import filelock
+    splits = split_path(path)
+    splits.basename = f".{splits.basename}"  # keep the lock file as a hidden file as it's not deleted by filelock on release
+    splits.extension = f"{splits.extension}.lock"
+    lock_path = path_from_split(splits, real_path=False)
+    with filelock.FileLock(lock_path, timeout=timeout):
+        yield
+
 
 def run_subprocess(*popenargs,
                    input=None, capture_output=False, timeout=None, check=False, communicate_fn=None,
@@ -102,6 +121,8 @@ def live_output_windows(process: subprocess.Popen, **_) -> Tuple[str, str]:
     )
 
     def forward_output(stream, queue_):
+        if isinstance(stream, io.TextIOWrapper):
+            stream.reconfigure(errors="ignore")
         for line in stream:
             queue_.put(line)
 
@@ -151,9 +172,20 @@ def read_pipe(pipe, timeout):
                 reads[i] = line
         return reads if len(pipes) > 1 else reads[0]
 
-    output, error = zip(*iter(lambda: read_pipe([process.stdout if process.stdout else 1,
-                                                 process.stderr if process.stderr else 2], activity_timeout),
-                              ['', '']))
+    process_output = list(iter(
+        lambda: read_pipe(
+            [process.stdout if process.stdout else 1,
+             process.stderr if process.stderr else 2
+             ], activity_timeout
+    ), ['', '']))
+    if not process_output:
+        log.warning(
+            "No framework process output detected, "
+            "this might indicate a problem with the logging configuration."
+        )
+        return '', ''
+
+    output, error = zip(*process_output)
     print()  # ensure that the log buffer is flushed at the end
     return ''.join(output), ''.join(error)
 
@@ -741,3 +773,4 @@ def profiler(*args, **kwargs):
     return decorator
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/serialization.py b/amlb/utils/serialization.py
index 6b554855c..dffa4c53c 100644
--- a/amlb/utils/serialization.py
+++ b/amlb/utils/serialization.py
@@ -10,6 +10,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def _import_data_libraries():
     try:
@@ -236,3 +238,4 @@ def deserialize_data(path, config: Optional[ns] = None):
         raise SerializationError(f"Can not deserialize file `{path}` in unknown format.")
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/time.py b/amlb/utils/time.py
index e73dcae6e..416688bcc 100644
--- a/amlb/utils/time.py
+++ b/amlb/utils/time.py
@@ -9,6 +9,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def datetime_iso(datetime=None, date=True, time=True, micros=False, date_sep='-', datetime_sep='T', time_sep=':', micros_sep='.', no_sep=False):
     """
@@ -160,3 +162,5 @@ def __exit__(self, *args):
         if self.timer:
             self.timer.cancel()
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/docs/.nojekyll b/docs/.nojekyll
new file mode 100644
index 000000000..e69de29bb
diff --git a/docs/Gemfile b/docs/Gemfile
deleted file mode 100644
index 24c316eba..000000000
--- a/docs/Gemfile
+++ /dev/null
@@ -1,3 +0,0 @@
-source 'https://rubygems.org'
-gem 'github-pages', group: :jekyll_plugins
-gem "jekyll-theme-hydeout", "~> 4"
diff --git a/docs/HOWTO.md b/docs/HOWTO.md
index 4276a0880..de72a14ea 100644
--- a/docs/HOWTO.md
+++ b/docs/HOWTO.md
@@ -145,7 +145,7 @@ Each dataset must contain a training set and a test set. There can be multiple t
 
 ### Datasets definition
 
-A dataset definition consists in a `yaml` file listing all the task/datasets that will used for the complete benchmark, 
+A dataset definition consists in a `yaml` file listing all the task/datasets that will be used for the complete benchmark, 
 or as an OpenML suite.
 
 Default dataset definitions are available under folder `resources/benchmarks`.
@@ -474,7 +474,7 @@ Among the existing frameworks, we can see different type of integrations:
 - Python API integration: these are frameworks that can be run directly from Python: cf. `autosklearn`, `H2OAutoML`, `TPOT`, `RandomForest`, `TunedRandomForest`.
    - contrary to the trivial integration, those require a `setup` phase.
    - Most of them currently run using the same dependencies as the application, which is not recommended due to potential version conflicts (especially with `sklearn`). This was not a major constraint with the first frameworks implemented, but now, those integrations can and will be slightly changed to [run in their dedicated virtual environment](#frameworks-requiring-a-dedicated-virtual-env), using their own dependencies: cf. `RandomForest` and `examples/custom/extensions/Stacking` for examples.
-- non-Python frameworks: those frameworks typically run in `R` or `Java` and don't provide any Python API. The integration is then still done by spawning the `R` or `Java` process from the `exec.py`: cf. `AutoWEKA`, `autoxgboost`.
+- non-Python frameworks: those frameworks typically run in `R` or `Java` and don't provide any Python API. The integration is then still done by spawning the `Java` or `R` process from the `exec.py`: cf. `AutoWEKA` or `ranger`, respectively.
 
 #### Recommended structure
 
@@ -619,18 +619,21 @@ frameworks/ranger/
 
 Here are the main differences:
 - the `setup` phase is identical, but if at runtime, some executable file or library is required that need to be installed locally (as opposed to globally: for example, `R` or `java` executable are usually installed globally), we just recommend to put everything under the integration module (for example in `lib` and/or `bin` subfolders as for `AutoWEKA`). This is also true for some Python frameworks (cf. `hyperoptsklearn` integration for example, where the modules are loaded from `frameworks/hyperoptsklearn/lib/hyperopt-sklearn`).
-- the framework is then executed by building a command manually in `exec.py`, runming it in a subprocess, and finally collecting the results generated by the subprocess. For example, in `autoxgboost/exec.py`:
+- the framework is then executed by building a command manually in `exec.py`, running it in a subprocess, and finally collecting the results generated by the subprocess. For example, in `ranger/exec.py`:
   ```python
   with Timer() as training:
-      run_cmd(r"""Rscript --vanilla -e "source('{script}'); run('{train}', '{test}', target.index = {target_index}, '{output}', {cores}, time.budget = {time_budget})" """.format(
-          script=os.path.join(here, 'exec.R'),
-          train=dataset.train.path,
-          test=dataset.test.path,
-          target_index=dataset.target.index+1,
-          output=config.output_predictions_file,
-          cores=config.cores,
-          time_budget=config.max_runtime_seconds
-      ), _live_output_=True)
+    run_cmd(("Rscript --vanilla -e \""
+             "source('{script}'); "
+             "run('{train}', '{test}', '{output}', cores={cores}, meta_results_file='{meta_results}', task_type='{task_type}')"
+             "\"").format(
+        script=os.path.join(here, 'exec.R'),
+        train=dataset.train.path,
+        test=dataset.test.path,
+        output=config.output_predictions_file,
+        meta_results=meta_results_file,
+        task_type=config.type,
+        cores=config.cores
+    ), _live_output_=True)
   ```
   Here, the `exec.R` script is also responsible to save the predictions in the expected format.
 
diff --git a/docs/README.md b/docs/README.md
index dc4e676ef..491eeb627 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -38,7 +38,7 @@ Documentation: <https://openml.github.io/automlbenchmark/>
 ## Installation
 ### Pre-requisites
 To run the benchmarks, you will need:
-* Python 3.7+.
+* Python 3.9+.
 * PIP3: ensure you have a recent version. If necessary, upgrade your pip using `python -m pip install -U pip`.
 * The Python libraries listed in [requirements.txt](../requirements.txt): it is strongly recommended to first create a [Python virtual environment](https://docs.python.org/3/library/venv.html#venv-def) (cf. also [Pyenv](https://github.com/pyenv/pyenv): quick install using `curl https://pyenv.run | bash` or `brew install pyenv`) and work in it if you don't want to mess up your global Python environment.
 * [Docker](https://docs.docker.com/install/), if you plan to run the benchmarks in a container.
@@ -70,7 +70,7 @@ venv\Scripts\activate
 
 or using pyenv:
 ```bash
-pyenv install {python_version: 3.7.4}
+pyenv install {python_version: 3.9.16}
 pyenv virtualenv ve-automl
 pyenv local ve-automl
 ```
@@ -111,65 +111,74 @@ python3 runbenchmark.py --help
 ```text
 usage: runbenchmark.py [-h] [-m {local,aws,docker,singularity}]
                        [-t [task_id [task_id ...]]]
-                       [-f [fold_num [fold_num ...]]] [-i input_dir]
+                       [-f [fold_num ...]] [-i input_dir]
                        [-o output_dir] [-u user_dir] [-p parallel_jobs]
                        [-s {auto,skip,force,only}] [-k [true|false]]
+                       [-e] [--logging LOGGING]
+                       [--openml-run-tag OPENML_RUN_TAG]
                        framework [benchmark] [constraint]
 
 positional arguments:
-  framework             The framework to evaluate as defined by default in
-                        resources/frameworks.yaml. To use a labelled framework
-                        (i.e. a framework defined in
-                        resources/frameworks_{label}.yaml), use the syntax
-                        {framework}:{label}.
-  benchmark             The benchmark type to run as defined by default in
-                        resources/benchmarks/{benchmark}.yaml, a path to a
-                        benchmark description file, or an openml suite or
-                        task. OpenML references should be formatted as
-                        'openml/s/X' and 'openml/t/Y', for studies and tasks
-                        respectively. Defaults to `test`.
-  constraint            The constraint definition to use as defined by default
-                        in resources/constraints.yaml. Defaults to `test`.
+  framework             The framework to evaluate as defined by default in resources/frameworks.yaml.
+                        To use a labelled framework (i.e. a framework defined in resources/frameworks-{label}.yaml),
+                        use the syntax {framework}:{label}.
+  benchmark             The benchmark type to run as defined by default in resources/benchmarks/{benchmark}.yaml,
+                        a path to a benchmark description file, or an openml suite or task.
+                        OpenML references should be formatted as 'openml/s/X' and 'openml/t/Y',
+                        for studies and tasks respectively. Use 'test.openml/s/X' for the 
+                        OpenML test server.
+                        (default: 'test')
+  constraint            The constraint definition to use as defined by default in resources/constraints.yaml.
+                        (default: 'test')
 
 optional arguments:
   -h, --help            show this help message and exit
-  -m {local,docker,aws,singularity}, --mode {local,docker,aws,singularity}
-                        The mode that specifies how/where the benchmark tasks
-                        will be running. Defaults to local.
-  -t [task_id [task_id ...]], --task [task_id [task_id ...]]
-                        The specific task name (as defined in the benchmark
-                        file) to run. When an OpenML reference is used as
-                        benchmark, the dataset name should be used instead. If
-                        not provided, then all tasks from the benchmark will
-                        be run.
-  -f [fold_num [fold_num ...]], --fold [fold_num [fold_num ...]]
-                        If task is provided, the specific fold(s) to run. If
-                        fold is not provided, then all folds from the task
-                        definition will be run.
+  -m {local,aws,docker,singularity}, --mode {local,aws,docker,singularity}
+                        The mode that specifies how/where the benchmark tasks will be running.
+                        (default: 'local')
+  -t [task_id ...], --task [task_id ...]
+                        The specific task name (as defined in the benchmark file) to run.
+                        When an OpenML reference is used as benchmark, the dataset name should be used instead.
+                        If not provided, then all tasks from the benchmark will be run.
+  -f [fold_num ...], --fold [fold_num ...]
+                        If task is provided, the specific fold(s) to run.
+                        If fold is not provided, then all folds from the task definition will be run.
   -i input_dir, --indir input_dir
-                        Folder where datasets are loaded by default. Defaults
-                        to `input_dir` as defined in resources/config.yaml
+                        Folder from where the datasets are loaded by default.
+                        (default: '~/.openml')
   -o output_dir, --outdir output_dir
-                        Folder where all the outputs should be written.
-                        Defaults to `output_dir` as defined in
-                        resources/config.yaml
+                        Folder where all the outputs should be written.(default: './results')
   -u user_dir, --userdir user_dir
-                        Folder where all the customizations are stored.
-                        Defaults to `user_dir` as defined in
-                        resources/config.yaml
+                        Folder where all the customizations are stored.(default: '~/.config/automlbenchmark')
   -p parallel_jobs, --parallel parallel_jobs
-                        The number of jobs (i.e. tasks or folds) that can run
-                        in parallel. Defaults to 1. Currently supported only
-                        in docker and aws mode.
+                        The number of jobs (i.e. tasks or folds) that can run in parallel.
+                        A hard limit is defined by property `job_scheduler.max_parallel_jobs`
+                         in `resources/config.yaml`.
+                        Override this limit in your custom `config.yaml` file if needed.
+                        Supported only in aws mode or container mode (docker, singularity).
+                        (default: 1)
   -s {auto,skip,force,only}, --setup {auto,skip,force,only}
-                        Framework/platform setup mode. Defaults to auto.
-                        •auto: setup is executed only if strictly necessary.
-                        •skip: setup is skipped. •force: setup is always
-                        executed before the benchmark. •only: only setup is
-                        executed (no benchmark).
+                        Framework/platform setup mode. Available values are:
+                        • auto: setup is executed only if strictly necessary.
+                        • skip: setup is skipped.
+                        • force: setup is always executed before the benchmark.
+                        • only: only setup is executed (no benchmark).
+                        (default: 'auto')
   -k [true|false], --keep-scores [true|false]
-                        Set to true [default] to save/add scores in output
-                        directory.
+                        Set to true (default) to save/add scores in output directory.
+  -e, --exit-on-error   If set, terminates on the first task that does not complete with a model.
+  --logging LOGGING     Set the log levels for the 3 available loggers:
+                        • console
+                        • app: for the log file including only logs from amlb (.log extension).
+                        • root: for the log file including logs from libraries (.full.log extension).
+                        Accepted values for each logger are: notset, debug, info, warning, error, fatal, critical.
+                        Examples:
+                          --logging=info (applies the same level to all loggers)
+                          --logging=root:debug (keeps defaults for non-specified loggers)
+                          --logging=console:warning,app:info
+                        (default: 'console:info,app:debug,root:info')
+  --openml-run-tag OPENML_RUN_TAG
+                        Tag that will be saved in metadata and OpenML runs created during upload, must match '([a-zA-Z0-9_\-\.])+'.
 ```
 
 The script will produce output that records task metadata and the result.
@@ -327,6 +336,32 @@ By default, a benchmark run creates the following subdirectories and files in th
 * `logs`: this subdirectory contains logs produced by the `automlbenchmark` app, including when it's been run in Docker container or on AWS.
 
 
+### Uploading results to OpenML
+The `upload_results.py` script can be used to upload results to OpenML with the following usage:
+```text
+>python upload_results.py --help
+usage: Script to upload results from the benchmark to OpenML. [-h] [-i INPUT_DIRECTORY] [-a APIKEY] [-m MODE] [-x] [-v] [-t TASK]
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -i INPUT_DIRECTORY, --input-directory INPUT_DIRECTORY
+                        Directory that stores results from the runbenchmark.py invocation. By default use the most recent folder in the results folder as
+                        specified in the configuration.
+  -a APIKEY, --api-key APIKEY
+                        OpenML API key to use for uploading results.
+  -m MODE, --mode MODE  Run mode (default=check).
+                        • check: only report whether results can be uploaded.
+                        • upload: upload all complete results.
+  -x, --fail-fast       Stop as soon as a task fails to upload due to an error during uploading.
+  -v, --verbose         Output progress to console.
+  -t TASK, --task TASK  Only upload results for this specific task.
+```
+
+Note that the default behavior does not upload data but only verifies data is complete.
+We strongly encourage you to only upload your data after verifying all expected results are complete.
+The OpenML Python package is used for uploading results, so to ensure your API credentials are configured, please refer to their [configuration documentation](https://openml.github.io/openml-python/master/usage.html#installation-set-up).
+Results obtained on tasks on the test server (e.g. through the `--test-server` option of `runbenchmark.py`) are uploaded to the test server and don't require additional authentication.
+
 ## Advanced configuration
 If you need to create your own benchmark, add a framework, create a plugin for a proprietary framework, or simply want to use some advanced options (e.g. run some frameworks with non-default parameters), see the [HOWTO].
 
diff --git a/docs/_config.yml b/docs/_config.yml
deleted file mode 100644
index f5bc67e41..000000000
--- a/docs/_config.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-repository: openml/automlbenchmark
-#theme: jekyll-theme-slate
-remote_theme: fongandrew/hydeout
-markdown: kramdown
-
-title: AutoML Benchmark
-#description: config description
-
-#sidebar_home_link: true
-version: 0.9
-
-hydeout:
-  repo: https://github.com/openml/automlbenchmark
-  version: 0.9
-
-defaults:
-  -
-    scope:
-      path: README.md
-    values:
-      title: ''
-  -
-    scope:
-      path: HOWTO.md
-    values:
-      title: ''
diff --git a/docs/assets/css/main.scss b/docs/assets/css/main.scss
deleted file mode 100644
index 61a51042f..000000000
--- a/docs/assets/css/main.scss
+++ /dev/null
@@ -1,16 +0,0 @@
----
-# Jekyll needs front matter for SCSS files
-# $sidebar-bg-color: #ac4142;
-# $sidebar-sticky: false;
----
-
-$sidebar-width: 20rem !default;
-$large-font-size: 1.1rem;
-$link-color: #195cc6;
-@import "hydeout";
-
-#sidebar {
-    .site-title {
-		font-family: Sans-Serif;
-    }
-}
\ No newline at end of file
diff --git a/docs/automl_overview.html b/docs/automl_overview.html
new file mode 100644
index 000000000..e72f07c04
--- /dev/null
+++ b/docs/automl_overview.html
@@ -0,0 +1,17 @@
+<html>
+  <head>
+    <title>AMLB</title>
+    <meta charset="UTF-8" />
+    <meta http-equiv="refresh" content="3; URL=frameworks.html" />
+  </head>
+  <body>
+    <p>
+      The old AutoML framework overview page no longer exists, you will be
+      redirected to the new page. If you are not redirected within 3 seconds,
+      you can find it at
+      <a href="https://openml.github.io/automlbenchmark/frameworks.html"
+        >https://openml.github.io/automlbenchmark/frameworks.html</a
+      >.
+    </p>
+  </body>
+</html>
diff --git a/docs/benchmark_datasets.html b/docs/benchmark_datasets.html
new file mode 100644
index 000000000..f9b28623f
--- /dev/null
+++ b/docs/benchmark_datasets.html
@@ -0,0 +1,18 @@
+<html>
+  <head>
+    <title>AMLB</title>
+    <meta charset="UTF-8" />
+    <meta http-equiv="refresh" content="5; URL=https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md"/>
+  </head>
+  <body>
+    <p>
+      You probably found this link from our 2019 paper.
+      Unfortunately, we updated our site but the new website does not yet contain a page with a description of our dataset selection strategy.<br/>
+        You will be redirected to the markdown file from which the old page was generated, so you can still view the old text.<br/>
+        If you are not redirected within 5 seconds, you can find it at
+      <a href="https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md"
+        >https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md</a
+      >.
+    </p>
+  </body>
+</html>
diff --git a/docs/frameworks.html b/docs/frameworks.html
new file mode 100644
index 000000000..98f48c9d4
--- /dev/null
+++ b/docs/frameworks.html
@@ -0,0 +1,989 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Frameworks</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <h1 class="page-title">AutoML Frameworks</h1>
+        There is more to an AutoML system than just its performance. This page
+        contains more information about the integrated AutoML frameworks,
+        including links to their papers, repositories
+        <img src="img/logos/GitHub-Mark-64px.png" height="16px" />, and
+        documentation 📖. Summaries taken directly from the respective
+        documentation pages. Want to integrate your own framework?
+        <a
+          href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-an-automl-framework"
+          target="_blank"
+          >Adding your own framework</a
+        >
+        is relatively simple.
+        <!-- 
+        <br />
+        <br />
+        An AutoML framework may only be available through an API for a specific
+        programming language, while others can work stand-alone. Some systems
+        might output models which can be used without further dependency on the
+        AutoML package, in other cases the AutoML system is still required to
+        use the model. Some systems might be developed with a specific domain in
+        mind. When choosing an AutoML system, it is essential to consider things
+        that are important to you. -->
+      </section>
+      <section class="framework-card-list">
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/autogluon.png" height="32px" />
+            <h3>AutoGluon</h3>
+            <div class="framework-links">
+              <a href="https://github.com/awslabs/autogluon" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://auto.gluon.ai/stable/index.html" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            AutoGluon enables easy-to-use and easy-to-extend AutoML with a focus
+            on automated stack ensembling, deep learning, and real-world
+            applications spanning image, text, and tabular data.
+          </div>
+          <input type="checkbox" id="more-autogluon" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                AutoGluon-Tabular: Robust and Accurate AutoML for Structured
+                Data
+              </h3>
+              <div class="paper-authors">
+                Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang,
+                Pedro Larroy, Mu Li, Alexander Smola
+              </div>
+              <div class="paper-abstract">
+                We introduce AutoGluon-Tabular, an open-source AutoML framework
+                that requires only a single line of Python to train highly
+                accurate machine learning models on an unprocessed tabular
+                dataset such as a CSV file. Unlike existing AutoML frameworks
+                that primarily focus on model/hyperparameter selection,
+                AutoGluon-Tabular succeeds by ensembling multiple models and
+                stacking them in multiple layers. Experiments reveal that our
+                multi-layer combination of many models offers better use of
+                allocated training time than seeking out the best. A second
+                contribution is an extensive evaluation of public and commercial
+                AutoML platforms including TPOT, H2O, AutoWEKA, auto-sklearn,
+                AutoGluon, and Google AutoML Tables. Tests on a suite of 50
+                classification and regression tasks from Kaggle and the OpenML
+                AutoML Benchmark reveal that AutoGluon is faster, more robust,
+                and much more accurate. We find that AutoGluon often even
+                outperforms the best-in-hindsight combination of all of its
+                competitors. In two popular Kaggle competitions, AutoGluon beat
+                99% of the participating data scientists after merely 4h of
+                training on the raw data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2020</strong>
+                  <div>ICML 2020 AutoML Workshop</div>
+                </div>
+                <a href="https://arxiv.org/pdf/2003.06505.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a href="https://arxiv.org/abs/2003.06505" target="_blank"
+                  >arxiv</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-autogluon">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/auto-sklearn.png" height="32px" />
+            <h3>Auto-sklearn</h3>
+            <div class="framework-links">
+              <a href="https://github.com/automl/auto-sklearn" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://automl.github.io/auto-sklearn/master/#"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            Auto-sklearn is an automated machine learning toolkit and a drop-in
+            replacement for a scikit-learn estimator. Auto-sklearn frees a
+            machine learning user from algorithm selection and hyperparameter
+            tuning. It leverages recent advantages in Bayesian optimization,
+            meta-learning and ensemble construction.
+          </div>
+          <input type="checkbox" id="more-askl" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
+              </h3>
+              <div class="paper-authors">
+                Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius
+                Lindauer, Frank Hutter
+              </div>
+              <div class="paper-abstract">
+                Automated Machine Learning (AutoML) supports practitioners and
+                researchers with the tedious task of designing machine learning
+                pipelines and has recently achieved substantial success. In this
+                paper we introduce new AutoML approaches motivated by our
+                winning submission to the second ChaLearn AutoML challenge. We
+                develop PoSH Auto-sklearn, which enables AutoML systems to work
+                well on large datasets under rigid time limits using a new,
+                simple and meta-feature-free meta-learning technique and employs
+                a successful bandit strategy for budget allocation. However,
+                PoSH Auto-sklearn introduces even more ways of running AutoML
+                and might make it harder for users to set it up correctly.
+                Therefore, we also go one step further and study the design
+                space of AutoML itself, proposing a solution towards truly
+                hands-free AutoML. Together, these changes give rise to the next
+                generation of our AutoML system, Auto-sklearn 2.0 . We verify
+                the improvements by these additions in a large experimental
+                study on 39 AutoML benchmark datasets and conclude the paper by
+                comparing to other popular AutoML frameworks and Auto-sklearn
+                1.0 , reducing the relative error by up to a factor of 4.5, and
+                yielding a performance in 10 minutes that is substantially
+                better than what Auto-sklearn 1.0 achieves within an hour.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>arXiv</div>
+                </div>
+                <a href="https://arxiv.org/pdf/2007.04074.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a href="https://arxiv.org/abs/2007.04074" target="_blank"
+                  >arxiv</a
+                >
+              </div>
+            </div>
+            <div class="paper">
+              <h3 class="paper-title">
+                Efficient and Robust Automated Machine Learning
+              </h3>
+              <div class="paper-authors">
+                Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost
+                Springenberg, Manuel Blum, Frank Hutter
+              </div>
+              <div class="paper-abstract">
+                The success of machine learning in a broad range of applications
+                has led to an ever-growing demand for machine learning systems
+                that can be used off the shelf by non-experts. To be effective
+                in practice, such systems need to automatically choose a good
+                algorithm and feature preprocessing steps for a new dataset at
+                hand, and also set their respective hyperparameters. Recent work
+                has started to tackle this automated machine learning (AutoML)
+                problem with the help of efficient Bayesian optimization
+                methods. In this work we introduce a robust new AutoML system
+                based on scikit-learn (using 15 classifiers, 14 feature
+                preprocessing methods, and 4 data preprocessing methods, giving
+                rise to a structured hypothesis space with 110 hyperparameters).
+                This system, which we dub auto-sklearn, improves on existing
+                AutoML methods by automatically taking into account past
+                performance on similar datasets, and by constructing ensembles
+                from the models evaluated during the optimization. Our system
+                won the first phase of the ongoing ChaLearn AutoML challenge,
+                and our comprehensive analysis on over 100 diverse datasets
+                shows that it substantially outperforms the previous state of
+                the art in AutoML. We also demonstrate the performance gains due
+                to each of our contributions and derive insights into the
+                effectiveness of the individual components of auto-sklearn.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2015</strong>
+                  <div>
+                    Advances in Neural Information Processing Systems 28 (NIPS
+                    2015)
+                  </div>
+                </div>
+                <a
+                  href="https://papers.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://papers.neurips.cc/paper/2015/hash/11d0e6287202fced83f79975ec59a3a6-Abstract.html"
+                  target="_blank"
+                  >neurips</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-askl">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/flaml.svg" height="32px" />
+            <h3>FLAML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/microsoft/FLAML" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://microsoft.github.io/FLAML/" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            FLAML is a lightweight Python library that finds accurate machine
+            learning models automatically, efficiently and economically. It
+            frees users from selecting learners and hyperparameters for each
+            learner.
+          </div>
+          <input type="checkbox" id="more-flaml" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                FLAML: A Fast and Lightweight AutoML Library
+              </h3>
+              <div class="paper-authors">
+                Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu
+              </div>
+              <div class="paper-abstract">
+                We study the problem of using low computational cost to automate
+                the choices of learners and hyperparameters for an ad-hoc
+                training dataset and error metric, by conducting trials of
+                different configurations on the given training data. We
+                investigate the joint impact of multiple factors on both trial
+                cost and model error, and propose several design guidelines.
+                Following them, we build a fast and lightweight library FLAML
+                which optimizes for low computational resource in finding
+                accurate models. FLAML integrates several simple but effective
+                search strategies into an adaptive system. It significantly
+                outperforms top-ranked AutoML libraries on a large open source
+                AutoML benchmark under equal, or sometimes orders of magnitude
+                smaller budget constraints.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>MLSys Conference 2021</div>
+                </div>
+                <a
+                  href="https://proceedings.mlsys.org/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-flaml">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/GAMA-icon.png" height="32px" />
+            <h3>GAMA</h3>
+            <div class="framework-links">
+              <a href="https://github.com/openml-labs/gama" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://openml-labs.github.io/gama/master/"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            GAMA is developed for AutoML research and features a flexible AutoML
+            pipeline, which makes it easy to develop and evaluate new AutoML
+            components. GAMA's benchmarking configuration features evolutionary
+            optimization and ensemble construction.
+          </div>
+          <input type="checkbox" id="more-gama" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                GAMA: A General Automated Machine Learning Assistant
+              </h3>
+              <div class="paper-authors">
+                Pieter Gijsbers, Joaquin Vanschoren
+              </div>
+
+              <!-- <div class="paper-year">2019</div>
+            <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+            <div class="paper-venue">cite</div> -->
+
+              <div class="paper-abstract">
+                The General Automated Machine learning Assistant (GAMA) is a
+                modular AutoML system developed to empower users to track and
+                control how AutoML algorithms search for optimal machine
+                learning pipelines, and facilitate AutoML research itself. In
+                contrast to current, often black-box systems, GAMA allows users
+                to plug in different AutoML and post-processing techniques, logs
+                and visualizes the search process, and supports easy
+                benchmarking. It currently features three AutoML search
+                algorithms, two model post-processing steps, and is designed to
+                allow for more components to be added.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>
+                    Joint European Conference on Machine Learning and Knowledge
+                    Discovery in Databases. Springer, Cham, 2020.
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/2007.04911.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-030-67670-4_39"
+                  target="_blank"
+                  >Springer</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-gama">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/h2o-automl-logo.jpeg" height="32px" />
+            <h3>H2O AutoML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/h2oai/h2o-3" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            H2O's AutoML can be used for automating the machine learning
+            workflow, which includes automatic training and tuning of many
+            models within a user-specified time-limit. H2O offers a number of
+            model explainability methods that apply to AutoML objects (groups of
+            models), as well as individual models (e.g. leader model).
+            Explanations can be generated automatically with a single function
+            call, providing a simple interface to exploring and explaining the
+            AutoML models.
+          </div>
+          <input type="checkbox" id="more-h2o" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                H2O AutoML: Scalable Automatic Machine Learning
+              </h3>
+              <div class="paper-authors">Erin LeDell and Sébastien Poirier</div>
+              <div class="paper-abstract">
+                H2O is an open source, distributed machine learning platform
+                designed to scale to very large datasets, with APIs in R,
+                Python, Java and Scala. We present H2O AutoML, a highly
+                scalable, fully-automated, supervised learning algorithm which
+                automates the pro- cess of training a large selection of
+                candidate models and stacked ensembles within a single function.
+                The result of the AutoML run is a “leaderboard”: a ranked list
+                of models, all of which can be easily exported for use in a
+                production environment. Models in the leader- board can be
+                ranked by numerous model performance metrics or other model
+                attributes such as training time or average per-row prediction
+                speed. The H2O AutoML algorithm relies on the efficient training
+                of H2O machine learning al- gorithms to produce a large number
+                of models in a short amount of time. H2O AutoML uses a
+                combination of fast random search and stacked ensembles to
+                achieve results competitive with, and often better than, other
+                frameworks which rely on more complex model tuning techniques
+                such as Bayesian optimization or genetic algorithms. H2O AutoML
+                trains a va- riety of algorithms (e.g. GBMs, Random Forests,
+                Deep Neural Networks, GLMs), yielding a healthy amount of
+                diversity across candidate models, which can be exploited by
+                stacked ensembles to produce a powerful final model. The
+                effectiveness of this technique is reflected in the OpenML
+                AutoML Benchmark, which compares the performance of several of
+                the most well known, open source AutoML systems across a number
+                of datasets.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2020</strong>
+                  <div>ICML 2020 AutoML Workshop</div>
+                </div>
+                <a
+                  href="https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-h2o">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/LightAutoML_logo_small.png" height="32px" />
+            <h3>LightAutoML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/sb-ai-lab/LightAutoML" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://lightautoml.readthedocs.io/en/latest/"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            LightAutoML is open-source Python library aimed at automated machine
+            learning. It is designed to be lightweight and efficient for various
+            tasks with tabular, text data.
+          </div>
+          <input type="checkbox" id="more-lama" class="accordion-input" />
+          <div class="accordion-content">
+            <div>Paper to be added.</div>
+            <div class="paper">
+              <!-- <h3 class="paper-title">
+                Automating biomedical data science through tree-based pipeline
+                optimization
+              </h3> -->
+              <div class="paper-authors">
+                Alexander Ryzhkov, Anton Vakhrushev, Dmitry Simakov, Vasilii
+                Bunakov, Rinchin Damdinov, Alexander Kirilin, Pavel Shvets
+              </div>
+              <!-- <div class="paper-abstract">
+                Automated machine learning (AutoML) systems are helpful data
+                science assistants designed to scan data for novel features,
+                select appropriate supervised learning models and optimize their
+                parameters. For this purpose, Tree-based Pipeline Optimization
+                Tool (TPOT) was developed using strongly typed genetic
+                programing (GP) to recommend an optimized analysis pipeline for
+                the data scientist’s prediction problem. However, like other
+                AutoML systems, TPOT may reach computational resource limits
+                when working on big data such as whole-genome expression data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2016</strong>
+                  <div>
+                    Applications of Evolutionary Computation, pages 123-137
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/1601.07925" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-319-31204-0_9"
+                  target="_blank"
+                  >springer</a
+                >
+              </div> -->
+            </div>
+          </div>
+          <label for="more-lama">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/mljar.png" height="32px" />
+            <h3>mljar-supervised</h3>
+            <div class="framework-links">
+              <a
+                href="https://github.com/mljar/mljar-supervised/"
+                target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://supervised.mljar.com" target="_blank">📖</a>
+            </div>
+          </div>
+          <div>
+            The mljar-supervised is an Automated Machine Learning Python package
+            that works with tabular data. It is designed to save time for a data
+            scientist. It abstracts the common way to preprocess the data,
+            construct the machine learning models, and perform hyper-parameters
+            tuning to find the best model 🏆. It is no black-box as you can see
+            exactly how the ML pipeline is constructed (with a detailed Markdown
+            report for each ML model).
+          </div>
+          <input type="checkbox" id="more-mljar" class="accordion-input" />
+          <div class="accordion-content">
+            <div>No paper available.</div>
+          </div>
+          <label for="more-mljar">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/tpot.jpeg" height="32px" />
+            <h3>TPOT</h3>
+            <div class="framework-links">
+              <a href="https://github.com/EpistasisLab/tpot" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="http://epistasislab.github.io/tpot/" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            TPOT is a Python Automated Machine Learning tool that optimizes
+            machine learning pipelines using genetic programming. It has a focus
+            on optimizing models for biomedical data.
+          </div>
+          <input type="checkbox" id="more-tpot" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                Automating biomedical data science through tree-based pipeline
+                optimization
+              </h3>
+              <div class="paper-authors">
+                Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A.
+                Lavender, La Creis Kidd, and Jason H. Moore
+              </div>
+              <div class="paper-abstract">
+                Automated machine learning (AutoML) systems are helpful data
+                science assistants designed to scan data for novel features,
+                select appropriate supervised learning models and optimize their
+                parameters. For this purpose, Tree-based Pipeline Optimization
+                Tool (TPOT) was developed using strongly typed genetic
+                programing (GP) to recommend an optimized analysis pipeline for
+                the data scientist’s prediction problem. However, like other
+                AutoML systems, TPOT may reach computational resource limits
+                when working on big data such as whole-genome expression data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2016</strong>
+                  <div>
+                    Applications of Evolutionary Computation, pages 123-137
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/1601.07925" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-319-31204-0_9"
+                  target="_blank"
+                  >springer</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-tpot">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/img/logos/GAMA-icon.png b/docs/img/logos/GAMA-icon.png
new file mode 100644
index 000000000..c9626aa28
Binary files /dev/null and b/docs/img/logos/GAMA-icon.png differ
diff --git a/docs/img/logos/GAMA.png b/docs/img/logos/GAMA.png
new file mode 100644
index 000000000..0ba100f0c
Binary files /dev/null and b/docs/img/logos/GAMA.png differ
diff --git a/docs/img/logos/GitHub-Mark-32px.png b/docs/img/logos/GitHub-Mark-32px.png
new file mode 100644
index 000000000..8b25551a9
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-32px.png differ
diff --git a/docs/img/logos/GitHub-Mark-64px.png b/docs/img/logos/GitHub-Mark-64px.png
new file mode 100644
index 000000000..182a1a3f7
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-64px.png differ
diff --git a/docs/img/logos/GitHub-Mark-Light-64px.png b/docs/img/logos/GitHub-Mark-Light-64px.png
new file mode 100644
index 000000000..73db1f61f
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-Light-64px.png differ
diff --git a/docs/img/logos/LightAutoML_logo_small.png b/docs/img/logos/LightAutoML_logo_small.png
new file mode 100644
index 000000000..8d268e390
Binary files /dev/null and b/docs/img/logos/LightAutoML_logo_small.png differ
diff --git a/docs/img/logos/auto-sklearn.png b/docs/img/logos/auto-sklearn.png
new file mode 100644
index 000000000..65141af67
Binary files /dev/null and b/docs/img/logos/auto-sklearn.png differ
diff --git a/docs/img/logos/autogluon.png b/docs/img/logos/autogluon.png
new file mode 100644
index 000000000..8afef59ab
Binary files /dev/null and b/docs/img/logos/autogluon.png differ
diff --git a/docs/img/logos/flaml.svg b/docs/img/logos/flaml.svg
new file mode 100644
index 000000000..5ae22b683
--- /dev/null
+++ b/docs/img/logos/flaml.svg
@@ -0,0 +1,1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1138.16 450.71"><defs><style>.cls-1{fill:#ff9406;}.cls-2{fill:#505d66;}</style></defs><g id="Layer_2" data-name="Layer 2"><g id="图层_1" data-name="图层 1"><path class="cls-1" d="M298,76.7,173.81.24a1.58,1.58,0,0,0-2.06,2.35L211.55,51a263.57,263.57,0,0,0-33.37,3.9,252.77,252.77,0,0,0-35,9,240.65,240.65,0,0,0-33,13.92,228.41,228.41,0,0,0-30.5,18.8,211.86,211.86,0,0,0-29,25.52,191.34,191.34,0,0,0-23,29.72,176.8,176.8,0,0,0-16.34,33.49,172.22,172.22,0,0,0-8.9,36.76L0,241a1.58,1.58,0,0,0,1.37,1.77,1.6,1.6,0,0,0,1-.22l79-47.9a1.55,1.55,0,0,0,.69-.86l1-3.16a145,145,0,0,1,26.41-47.86,170.28,170.28,0,0,1,41.5-36A196.9,196.9,0,0,1,203.12,84.1a214.83,214.83,0,0,1,59.32-7.55c2.86,0,5.77.12,8.63.27s5.76.34,8.62.6,5.75.57,8.6.93,5.71.78,8.54,1.25a1.58,1.58,0,0,0,1.91-1.16A1.59,1.59,0,0,0,298,76.7Z" transform="translate(0.01 0.01)"/><path class="cls-1" d="M347.83,177.83l-1-18.05a1.57,1.57,0,0,0-1.65-1.48,1.49,1.49,0,0,0-.79.26l-71.16,47.15a1.55,1.55,0,0,0-.67,1L271.9,210a143.76,143.76,0,0,1-22.58,52.27,174.42,174.42,0,0,1-42.61,43,205,205,0,0,1-58.31,28.88,217.42,217.42,0,0,1-68.28,9.93c-3.3-.05-6.63-.17-9.89-.36s-6.58-.47-9.83-.8-6.51-.76-9.73-1.24-6.42-1.05-9.6-1.68a1.57,1.57,0,0,0-1.3,2.76L171.15,450.34a1.57,1.57,0,0,0,2.39-1.94l-36.87-70.52a264,264,0,0,0,40.57-5.5A251.22,251.22,0,0,0,217.75,360a238,238,0,0,0,36.9-18.61,224.15,224.15,0,0,0,32.27-24.25,201.9,201.9,0,0,0,28.2-31.37,179.69,179.69,0,0,0,19.59-34.43A167,167,0,0,0,345.6,215,161.86,161.86,0,0,0,347.83,177.83Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M258.56,209.79,196.9,181.24l61.42-95.48a1.63,1.63,0,0,0-2.23-2.26L101.25,179.84a1.63,1.63,0,0,0-.52,2.25,1.56,1.56,0,0,0,.67.6l60.26,29.12-90.33,122a1.62,1.62,0,0,0,.13,2.09,1.6,1.6,0,0,0,2.08.23l185.24-123.5a1.63,1.63,0,0,0,.46-2.26,1.67,1.67,0,0,0-.68-.58Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M451.86,199a36.63,36.63,0,0,0-12.3,10.44,32.45,32.45,0,0,0-6.35,14.49l-4.09,24.49h104.4l-4,24H425.12l-8.88,53.25H380.3l3-17.79,14-84a60.11,60.11,0,0,1,11.49-26.51,67.08,67.08,0,0,1,22.41-19.21,58.25,58.25,0,0,1,27.69-7.07H546.4l-4,24H466.76A31.6,31.6,0,0,0,451.86,199Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M772.68,325.65,742.8,208.39l-46.3,78,60.5,2.68L637.67,325.65,729,171.09h39.76l39.83,154.56Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M643.32,301.39H597.81q-11.91,0-18.91-8.43t-5-20.33l16.93-101.54H554.61L537.69,272.63q-2.4,14.39,2.65,26.51a41.67,41.67,0,0,0,16.11,19.32,45.49,45.49,0,0,0,25.42,7.19H629Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M975.05,170.86h36.17l-25.8,154.79H949.25L966,225l-37.44,45.37H892.87l-23.7-46.5-17,101.77H816l25.8-154.79H878l37.1,72.57Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M1138.15,301.39h-75.68q-11.91,0-18.92-8.43t-5-20.33l16.92-101.54H1019.3l-16.93,101.54a47.91,47.91,0,0,0,2.63,26.51,41.7,41.7,0,0,0,16.1,19.32,45.57,45.57,0,0,0,25.42,7.19h77.29Z" transform="translate(0.01 0.01)"/></g></g></svg>
\ No newline at end of file
diff --git a/docs/img/logos/h2o-automl-logo.jpeg b/docs/img/logos/h2o-automl-logo.jpeg
new file mode 100644
index 000000000..68ab95b32
Binary files /dev/null and b/docs/img/logos/h2o-automl-logo.jpeg differ
diff --git a/docs/img/logos/mljar.png b/docs/img/logos/mljar.png
new file mode 100644
index 000000000..a61427bef
Binary files /dev/null and b/docs/img/logos/mljar.png differ
diff --git a/docs/img/logos/tpot.jpeg b/docs/img/logos/tpot.jpeg
new file mode 100644
index 000000000..fd4d56add
Binary files /dev/null and b/docs/img/logos/tpot.jpeg differ
diff --git a/docs/img/shiny.png b/docs/img/shiny.png
new file mode 100644
index 000000000..9942dfeec
Binary files /dev/null and b/docs/img/shiny.png differ
diff --git a/docs/index.html b/docs/index.html
new file mode 100644
index 000000000..c6f1a808d
--- /dev/null
+++ b/docs/index.html
@@ -0,0 +1,443 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: an AutoML Benchmark</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <div class="title">AMLB</div>
+        <div class="subtitle">An AutoML Benchmark</div>
+        <div class="summary">
+          Comparing different AutoML frameworks is notoriously challenging. AMLB
+          is an open and extensible benchmark that follows best practices and
+          avoids common mistakes when comparing AutoML frameworks.
+        </div>
+      </section>
+      <section>
+        <div class="three-cols">
+          <div class="card">
+            <h3>Easy to Use</h3>
+            <div class="grey">
+              You can run an entire benchmark with a single command! The AutoML
+              benchmark tool automates the installation of the AutoML framework,
+              the experimental setup, and executing the experiment.
+            </div>
+            <div class="terminal">
+              > python runbenchmark.py autosklearn openml/s/269 1h8c
+            </div>
+            <div class="flex-grow"></div>
+            <a
+              class="card-nav"
+              href="https://github.com/openml/automlbenchmark#installation"
+              target="_blank"
+            >
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Installation guide
+            </a>
+          </div>
+          <div class="card">
+            <h3>Visualize Results</h3>
+            <div class="grey">
+              The results can be visualized with our
+              <a
+                href="https://compstat-lmu.shinyapps.io/AutoML-Benchmark-Analysis/"
+                target="_blank"
+                >interactive visualization tool</a
+              >
+              or one of our
+              <a
+                href="https://github.com/openml/automlbenchmark/tree/master/reports"
+                >notebooks</a
+              >. This includes stripplots, critical difference diagrams,
+              Bradley-Terry trees, and more!
+            </div>
+            <img src="img/shiny.png" width="100%" />
+            <div class="flex-grow"></div>
+            <a class="card-nav" href="results.html" target="_blank">
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Results
+            </a>
+          </div>
+          <div class="card">
+            <h3>Easy to Extend</h3>
+            <div class="grey">
+              <a
+                href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-a-benchmark"
+                >Adding a framework</a
+              >
+              and
+              <a
+                href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-a-benchmark"
+                >adding a dataset</a
+              >
+              is easy. These extensions can be kept completely private, or
+              shared with the community. For datasets, it is even possible to
+              work with
+              <a href="https://www.openml.org" target="_blank">OpenML</a>
+              tasks and suites directly!
+            </div>
+            <svg width="72" height="72" viewBox="0, 0, 400,400">
+              <g>
+                <path
+                  id="path0"
+                  d="M77.077 264.657 C 76.925 264.809,76.800 265.251,76.800 265.639 C 76.800 266.066,76.445 266.395,75.900 266.472 C 75.017 266.598,74.998 266.717,74.890 272.700 L 74.779 278.800 75.790 278.800 C 76.496 278.800,76.800 279.017,76.800 279.520 C 76.800 280.994,77.584 281.200,83.200 281.200 C 88.816 281.200,89.600 280.994,89.600 279.520 C 89.600 279.017,89.904 278.800,90.610 278.800 L 91.621 278.800 91.510 272.700 C 91.403 266.769,91.376 266.597,90.525 266.475 C 89.979 266.398,89.602 266.021,89.525 265.475 C 89.399 264.591,77.899 263.834,77.077 264.657 M89.867 302.667 C 89.720 302.813,89.600 303.251,89.600 303.639 C 89.600 304.066,89.245 304.395,88.700 304.472 C 87.806 304.599,87.800 304.645,87.800 311.600 C 87.800 318.555,87.806 318.601,88.700 318.728 C 89.283 318.811,89.600 319.131,89.600 319.638 C 89.600 320.862,91.193 321.200,96.926 321.195 C 103.118 321.188,103.852 321.043,104.174 319.762 C 104.321 319.176,104.707 318.800,105.161 318.800 C 106.215 318.800,106.388 317.773,106.395 311.474 C 106.400 305.959,106.091 304.400,104.994 304.400 C 104.675 304.400,104.302 303.950,104.164 303.400 L 103.913 302.400 97.023 302.400 C 93.234 302.400,90.013 302.520,89.867 302.667 M281.200 322.140 C 281.200 322.984,280.968 323.128,279.226 323.359 C 277.588 323.576,277.231 323.772,277.126 324.510 C 277.046 325.075,276.675 325.446,276.110 325.526 C 275.372 325.631,275.176 325.988,274.959 327.626 C 274.728 329.368,274.584 329.600,273.740 329.600 L 272.784 329.600 272.892 337.900 C 272.998 346.023,273.018 346.203,273.851 346.321 C 274.543 346.420,274.751 346.809,274.962 348.395 C 275.176 350.012,275.373 350.369,276.110 350.474 C 276.675 350.554,277.046 350.925,277.126 351.490 C 277.231 352.228,277.588 352.424,279.226 352.641 C 280.968 352.872,281.200 353.016,281.200 353.860 L 281.200 354.816 289.500 354.708 C 297.766 354.600,297.801 354.596,297.928 353.700 C 298.019 353.056,298.325 352.800,299.002 352.800 C 300.639 352.800,302.000 352.078,302.000 351.210 C 302.000 350.663,302.263 350.400,302.810 350.400 C 303.678 350.400,304.400 349.039,304.400 347.402 C 304.400 346.725,304.656 346.419,305.300 346.328 C 306.682 346.132,306.729 329.875,305.349 329.679 C 304.655 329.580,304.449 329.191,304.235 327.579 C 304.017 325.933,303.834 325.600,303.149 325.600 C 302.651 325.600,302.239 325.271,302.108 324.769 C 301.875 323.878,300.533 323.200,299.002 323.200 C 298.325 323.200,298.019 322.944,297.928 322.300 C 297.801 321.404,297.766 321.400,289.500 321.292 L 281.200 321.184 281.200 322.140 M117.067 334.267 C 116.920 334.413,116.800 334.863,116.800 335.267 C 116.800 335.782,116.503 336.000,115.800 336.000 C 115.008 336.000,114.800 336.202,114.800 336.972 C 114.800 337.676,114.551 337.980,113.900 338.072 C 113.010 338.199,112.999 338.280,112.891 345.290 L 112.782 352.380 113.768 352.628 C 114.371 352.779,114.802 353.210,114.877 353.738 C 114.952 354.265,115.335 354.648,115.862 354.723 C 116.390 354.798,116.821 355.229,116.972 355.832 L 117.220 356.818 124.310 356.709 C 131.320 356.601,131.401 356.590,131.528 355.700 C 131.620 355.049,131.924 354.800,132.628 354.800 C 133.391 354.800,133.600 354.591,133.600 353.828 C 133.600 353.124,133.849 352.820,134.500 352.728 C 135.391 352.601,135.401 352.528,135.509 345.300 L 135.618 338.000 134.609 338.000 C 133.799 338.000,133.600 337.802,133.600 337.000 C 133.600 336.200,133.400 336.000,132.600 336.000 C 131.800 336.000,131.600 335.800,131.600 335.000 L 131.600 334.000 124.467 334.000 C 120.543 334.000,117.213 334.120,117.067 334.267 M245.680 348.880 C 245.416 349.144,245.200 349.684,245.200 350.080 C 245.200 350.542,244.903 350.800,244.369 350.800 C 243.308 350.800,241.200 352.890,241.200 353.941 C 241.200 354.470,240.892 354.788,240.300 354.872 C 239.400 355.000,239.400 355.001,239.400 363.200 C 239.400 371.399,239.400 371.400,240.300 371.528 C 240.892 371.612,241.200 371.930,241.200 372.459 C 241.200 373.510,243.308 375.600,244.369 375.600 C 244.903 375.600,245.200 375.858,245.200 376.320 C 245.200 377.855,245.910 378.000,253.447 378.000 C 261.546 378.000,262.000 377.923,262.000 376.557 C 262.000 375.773,262.205 375.600,263.132 375.600 C 263.931 375.600,264.423 375.305,264.800 374.600 C 265.094 374.050,265.575 373.600,265.868 373.600 C 266.165 373.600,266.400 373.170,266.400 372.628 C 266.400 371.924,266.649 371.620,267.300 371.528 C 268.200 371.400,268.200 371.399,268.200 363.200 C 268.200 355.001,268.200 355.000,267.300 354.872 C 266.749 354.794,266.400 354.466,266.400 354.028 C 266.400 353.634,265.950 352.974,265.400 352.561 C 264.850 352.148,264.400 351.583,264.400 351.305 C 264.400 351.011,263.900 350.800,263.200 350.800 C 262.201 350.800,262.000 350.640,262.000 349.843 C 262.000 349.316,261.715 348.776,261.367 348.643 C 260.116 348.163,246.187 348.373,245.680 348.880 M155.288 353.272 C 155.020 353.540,154.800 353.981,154.800 354.252 C 154.800 354.523,154.395 354.802,153.900 354.872 C 153.262 354.963,152.964 355.313,152.876 356.075 C 152.789 356.828,152.489 357.188,151.876 357.275 C 150.474 357.475,150.498 373.329,151.900 373.528 C 152.551 373.620,152.800 373.924,152.800 374.628 C 152.800 375.386,153.010 375.600,153.757 375.600 C 154.430 375.600,154.784 375.881,154.951 376.549 C 155.266 377.803,156.387 378.000,163.200 378.000 C 170.013 378.000,171.134 377.803,171.449 376.549 C 171.616 375.881,171.970 375.600,172.643 375.600 C 173.390 375.600,173.600 375.386,173.600 374.628 C 173.600 373.924,173.849 373.620,174.500 373.528 C 175.394 373.401,175.401 373.351,175.508 365.493 L 175.617 357.587 174.608 357.203 C 173.898 356.933,173.600 356.521,173.600 355.810 C 173.600 355.028,173.396 354.800,172.695 354.800 C 172.140
+ 354.800,171.637 354.452,171.395 353.900 C 170.923 352.824,156.306 352.254,155.288 353.272 M203.476 357.058 C 203.324 357.209,203.200 357.743,203.200 358.245 C 203.200 359.048,202.975 359.171,201.300 359.278 L 199.400 359.400 199.279 361.275 C 199.169 362.979,199.059 363.162,198.079 363.275 L 197.000 363.400 197.000 370.600 L 197.000 377.800 198.079 377.925 C 199.059 378.038,199.169 378.221,199.279 379.925 L 199.400 381.800 201.275 381.921 C 202.979 382.031,203.162 382.141,203.275 383.121 L 203.400 384.200 210.600 384.200 L 217.800 384.200 217.925 383.121 C 218.038 382.141,218.221 382.031,219.925 381.921 L 221.800 381.800 221.921 379.925 C 222.031 378.221,222.141 378.038,223.121 377.925 L 224.200 377.800 224.200 370.600 L 224.200 363.400 223.121 363.275 C 222.141 363.162,222.031 362.979,221.921 361.275 L 221.800 359.400 219.925 359.279 C 218.221 359.169,218.038 359.059,217.925 358.079 L 217.800 357.000 210.776 356.891 C 206.912 356.831,203.627 356.906,203.476 357.058 "
+                  stroke="none"
+                  fill="#34cc64"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path1"
+                  d="M203.467 47.867 C 203.320 48.013,203.200 48.451,203.200 48.839 C 203.200 49.266,202.845 49.595,202.300 49.672 C 201.417 49.798,201.398 49.917,201.290 55.900 L 201.179 62.000 202.190 62.000 C 202.799 62.000,203.200 62.227,203.200 62.571 C 203.200 64.074,204.305 64.400,209.396 64.400 C 215.031 64.400,215.600 64.270,215.600 62.984 C 215.600 62.266,215.851 62.024,216.700 61.927 L 217.800 61.800 217.871 55.884 L 217.942 49.968 216.771 49.734 C 215.871 49.554,215.600 49.280,215.600 48.550 L 215.600 47.600 209.667 47.600 C 206.403 47.600,203.613 47.720,203.467 47.867 M176.276 87.858 C 176.124 88.009,176.000 88.463,176.000 88.867 C 176.000 89.270,175.820 89.623,175.600 89.650 C 175.380 89.678,174.975 89.723,174.700 89.750 C 174.324 89.788,174.181 91.551,174.125 96.863 L 174.050 103.926 175.025 104.171 C 175.910 104.393,176.062 104.604,175.970 105.484 C 175.903 106.130,177.702 106.344,183.200 106.344 C 188.698 106.344,190.497 106.130,190.430 105.484 C 190.338 104.605,190.490 104.393,191.365 104.173 C 193.344 103.676,193.472 89.799,191.500 89.626 C 191.335 89.612,191.033 89.600,190.828 89.600 C 190.623 89.600,190.398 89.195,190.328 88.700 C 190.199 87.794,177.123 87.010,176.276 87.858 M157.308 125.736 C 157.017 126.193,156.949 126.410,156.850 127.200 C 156.822 127.420,156.350 127.600,155.800 127.600 C 155.028 127.600,154.800 127.806,154.800 128.505 C 154.800 129.060,154.452 129.563,153.900 129.805 C 153.023 130.190,152.997 130.378,152.891 137.090 L 152.781 143.980 153.791 144.233 C 154.420 144.391,154.800 144.769,154.800 145.236 C 154.800 145.694,155.175 146.079,155.765 146.227 C 156.344 146.372,156.829 146.860,156.975 147.444 L 157.220 148.419 164.110 148.309 C 170.822 148.203,171.010 148.177,171.395 147.300 C 171.637 146.748,172.140 146.400,172.695 146.400 C 173.394 146.400,173.600 146.172,173.600 145.400 C 173.600 144.850,173.780 144.378,174.000 144.350 C 174.220 144.322,174.625 144.278,174.900 144.250 C 175.275 144.212,175.427 142.382,175.509 136.900 L 175.618 129.600 174.609 129.600 C 173.799 129.600,173.600 129.402,173.600 128.600 C 173.600 127.800,173.400 127.600,172.600 127.600 C 171.812 127.600,171.600 127.397,171.600 126.643 C 171.600 126.115,171.298 125.570,170.928 125.428 C 169.703 124.958,157.629 125.231,157.308 125.736 M146.892 174.607 C 146.612 174.925,146.393 175.323,146.404 175.492 C 146.415 175.661,145.969 175.857,145.412 175.927 C 144.583 176.030,144.400 176.269,144.400 177.250 C 144.400 177.952,144.233 178.344,143.997 178.198 C 142.725 177.412,142.400 179.078,142.400 186.400 C 142.400 193.722,142.725 195.388,143.997 194.602 C 144.219 194.465,144.400 194.687,144.400 195.096 C 144.400 196.048,145.152 196.800,146.104 196.800 C 146.513 196.800,146.735 196.981,146.598 197.203 C 145.812 198.475,147.478 198.800,154.800 198.800 C 162.122 198.800,163.788 198.475,163.002 197.203 C 162.859 196.972,163.194 196.800,163.786 196.800 C 164.614 196.800,164.868 196.562,165.059 195.606 C 165.190 194.950,165.457 194.512,165.652 194.632 C 166.873 195.387,167.200 193.646,167.200 186.400 C 167.200 179.078,166.875 177.412,165.603 178.198 C 165.367 178.344,165.200 177.952,165.200 177.250 C 165.200 176.242,165.027 176.033,164.100 175.927 C 163.495 175.857,163.051 175.659,163.114 175.486 C 163.671 173.954,148.206 173.116,146.892 174.607 M148.564 219.000 C 148.387 219.708,148.042 220.000,147.383 220.000 C 145.156 220.000,144.400 220.657,144.400 222.590 C 144.400 224.254,144.319 224.400,143.400 224.400 L 142.400 224.400 142.400 231.600 L 142.400 238.800 143.400 238.800 C 144.276 238.800,144.400 238.973,144.400 240.200 C 144.400 242.031,144.860 243.148,145.600 243.114 C 147.635 243.022,148.354 243.269,148.472 244.100 C 148.599 244.991,148.672 245.001,155.900 245.109 L 163.200 245.218 163.200 244.209 C 163.200 243.366,163.397 243.192,164.400 243.150 C 165.060 243.123,165.780 243.055,166.000 243.000 C 166.220 242.945,166.545 242.878,166.722 242.850 C 166.899 242.822,167.088 242.048,167.142 241.129 C 167.261 239.114,167.428 238.800,168.381 238.800 C 169.477 238.800,169.600 238.075,169.600 231.600 C 169.600 224.910,169.498 224.400,168.157 224.400 C 167.309 224.400,167.200 224.220,167.200 222.823 C 167.200 221.016,166.160 220.000,164.311 220.000 C 163.393 220.000,163.200 219.827,163.200 219.000 L 163.200 218.000 156.008 218.000 L 148.815 218.000 148.564 219.000 M168.100 258.265 C 167.825 258.376,167.600 258.812,167.600 259.233 C 167.600 259.740,167.321 260.000,166.780 260.000 C 165.687 260.000,163.200 262.459,163.200 263.540 C 163.200 264.117,162.951 264.400,162.443 264.400 C 162.026 264.400,161.576 264.685,161.443 265.033 C 161.019 266.136,161.165 280.796,161.600 280.850 C 161.820 280.877,162.223 280.923,162.495 280.950 C 162.767 280.978,163.056 281.360,163.138 281.800 C 163.327 282.821,163.995 283.650,164.418 283.389 C 164.598 283.278,164.875 283.398,165.034 283.655 C 165.215 283.948,165.150 284.016,164.862 283.838 C 164.215 283.439,164.285 283.942,164.971 284.629 C 165.286 284.943,166.006 285.200,166.571 285.200 C 167.402 285.200,167.600 285.393,167.600 286.200 L 167.600 287.200 175.576 287.200 C 183.535 287.200,184.174 287.096,184.194 285.800 C 184.197 285.580,184.695 285.343,185.300 285.273 C 185.905 285.204,186.400 284.955,186.400 284.720 C 186.400 284.485,186.833 283.912,187.362 283.447 C 187.891 282.981,188.388 282.240,188.467 281.800 C 188.546 281.360,188.788 280.997,189.005 280.994 C 190.292 280.974,190.400 280.323,190.400 272.584 L 190.400 264.815 189.400 264.564 C 188.793 264.412,188.400 264.028,188.400 263.587 C 188.400 262.718,186.246 260.474,184.931 259.974 C 184.416 259.778,184.017 259.479,184.044 259.309 C 184.250 258.046,183.974 258.000,176.300 258.032 C 172.065 258.049,168.375 258.154,168.100 258.265 M201.200 282.148 C 201.200 283.108,200.906 283.235,197.700 283.666 C 197.425 283.703,197.284 283.817,197.386 283.919 C 197.790 284.323,196.726 285.600,195.986 285.600 C 195.554 285.600,195.178 285.780,195.150 286.000 C 195.123 286.220,195.055 286.580,195.000 286.800 C 194.945 287.020,194.878 287.740,194.850 288.400 C 194.808 289.403,194.634 289.600,193.792 289.600 L 192.784 289.600 192.892 297.900 C 193.000 306.166,193.004 306.201,193.900 306.328 C 194.585 306.425,194.812 306.736,194.850 307.628 C 194.878 308.273,194.945 308.980,195.000 309.200 C 195.055 309.420,195.123 309.791,195.150 310.024 C 195.178 310.257,195.373 310.340,195.585 310.209 C 196.048 309.923,197.714 311.492,197.432 311.948 C 197.324 312.123,197.452 312.297,197.718 312.334 C 200.960 312.786,201.200 312.890,201.200 313.852 L 201.200 314.816 209.500 314.708 C 217.766 314.600,217.801 314.596,217.928 313.700 C 218.020 313.054,218.325 312.800,219.009 312.800 C 220.477 312.800,222.000 311.953,222.000 311.136 C 222.000 310.688,222.288 310.400,222.736 310.400 C 223.559 310.400,224.400 308.874,224.400 307.381 C 224.400 306.842,224.580 306.377,224.800 306.350 C 225.020 306.322,225.425 306.278,225.700 306.250 C 226.736 306.146,226.382 289.826,225.341 289.678 C 224.521 289.561,224.336 289.090,223.934 286.100 C 223.897 285.825,223.783 285.684,223.681 285.786 C 223.356 286.110,222.000 285.194,222.000 284.650 C 222.000 284.016,220.316 283.200,219.009 283.200 C 218.325 283.200,218.020 282.946,217.928 282.300 C 217.801 281.404,217.766 281.400,209.500 281.292 L 201.200 281.184 201.200 282.148 "
+                  stroke="none"
+                  fill="#fccc04"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path2"
+                  d="M289.877 3.456 C 289.725 3.609,289.600 4.051,289.600 4.439 C 289.600 4.866,289.245 5.195,288.700 5.272 C 287.811 5.398,287.800 5.475,287.800 11.600 C 287.800 17.725,287.811 17.802,288.700 17.928 C 289.351 18.020,289.600 18.324,289.600 19.028 L 289.600 20.000 295.800 20.000 L 302.000 20.000 302.000 19.027 C 302.000 18.262,302.236 18.026,303.100 17.927 L 304.200 17.800 304.200 11.600 L 304.200 5.400 303.125 5.276 C 302.372 5.189,302.012 4.889,301.925 4.276 C 301.799 3.391,290.698 2.636,289.877 3.456 M256.276 31.058 C 256.124 31.209,256.000 31.651,256.000 32.039 C 256.000 32.466,255.645 32.795,255.100 32.872 C 254.206 32.999,254.200 33.045,254.200 40.000 C 254.200 46.955,254.206 47.001,255.100 47.128 C 255.737 47.218,256.000 47.526,256.000 48.181 C 256.000 49.438,256.775 49.589,263.200 49.589 C 269.659 49.589,270.400 49.441,270.400 48.153 C 270.400 47.416,270.616 47.200,271.353 47.200 C 272.644 47.200,272.788 46.467,272.795 39.874 C 272.800 34.138,272.501 32.800,271.214 32.800 C 270.732 32.800,270.409 32.472,270.328 31.900 C 270.199 30.994,257.123 30.210,256.276 31.058 M228.676 64.658 C 228.524 64.809,228.400 65.263,228.400 65.667 C 228.400 66.182,228.103 66.400,227.400 66.400 C 226.608 66.400,226.400 66.602,226.400 67.372 C 226.400 68.076,226.151 68.380,225.500 68.472 C 224.605 68.599,224.600 68.638,224.600 75.800 C 224.600 82.962,224.605 83.001,225.500 83.128 C 226.151 83.220,226.400 83.524,226.400 84.228 C 226.400 84.991,226.609 85.200,227.372 85.200 C 228.076 85.200,228.380 85.449,228.472 86.100 C 228.599 86.991,228.672 87.001,235.900 87.109 L 243.200 87.218 243.200 86.209 C 243.200 85.399,243.398 85.200,244.200 85.200 C 245.000 85.200,245.200 85.000,245.200 84.200 C 245.200 83.398,245.399 83.200,246.209 83.200 L 247.218 83.200 247.109 75.900 C 247.001 68.672,246.991 68.599,246.100 68.472 C 245.449 68.380,245.200 68.076,245.200 67.372 C 245.200 66.609,244.991 66.400,244.228 66.400 C 243.524 66.400,243.220 66.151,243.128 65.500 C 242.999 64.595,229.524 63.809,228.676 64.658 M203.600 103.400 C 203.333 104.102,202.919 104.400,202.210 104.400 C 201.412 104.400,201.200 104.600,201.200 105.357 C 201.200 106.043,200.918 106.384,200.200 106.564 L 199.200 106.815 199.200 114.800 L 199.200 122.785 200.200 123.036 C 200.918 123.216,201.200 123.557,201.200 124.243 C 201.200 125.000,201.412 125.200,202.210 125.200 C 202.919 125.200,203.333 125.498,203.600 126.200 L 203.980 127.200 211.782 127.200 L 219.585 127.200 219.836 126.200 C 220.016 125.482,220.357 125.200,221.043 125.200 C 221.797 125.200,222.000 124.988,222.000 124.200 C 222.000 123.463,222.213 123.200,222.810 123.200 C 224.102 123.200,224.400 121.627,224.400 114.800 C 224.400 107.973,224.102 106.400,222.810 106.400 C 222.213 106.400,222.000 106.137,222.000 105.400 C 222.000 104.612,221.797 104.400,221.043 104.400 C 220.357 104.400,220.016 104.118,219.836 103.400 L 219.585 102.400 211.782 102.400 L 203.980 102.400 203.600 103.400 M188.564 147.400 C 188.399 148.057,188.034 148.400,187.500 148.400 C 186.381 148.400,184.400 150.570,184.400 151.796 C 184.400 152.602,184.202 152.800,183.392 152.800 L 182.384 152.800 182.492 161.100 C 182.600 169.366,182.604 169.401,183.500 169.528 C 184.157 169.621,184.400 169.924,184.400 170.646 C 184.400 171.921,186.079 173.600,187.354 173.600 C 188.076 173.600,188.379 173.843,188.472 174.500 C 188.599 175.396,188.634 175.400,196.900 175.508 L 205.200 175.616 205.200 174.608 C 205.200 173.826,205.404 173.600,206.111 173.600 C 207.261 173.600,209.600 171.499,209.600 170.466 C 209.600 169.967,209.962 169.596,210.608 169.434 L 211.617 169.181 211.508 161.090 C 211.401 153.042,211.395 152.999,210.500 152.872 C 209.904 152.788,209.600 152.470,209.600 151.932 C 209.600 150.867,207.114 148.400,206.040 148.400 C 205.411 148.400,205.200 148.149,205.200 147.400 L 205.200 146.400 197.008 146.400 L 188.815 146.400 188.564 147.400 M188.667 191.067 C 188.520 191.213,188.400 191.663,188.400 192.067 C 188.400 192.570,188.105 192.800,187.458 192.800 C 186.186 192.800,184.400 194.524,184.400 195.753 C 184.400 196.477,184.157 196.779,183.500 196.872 C 182.604 196.999,182.600 197.034,182.492 205.300 L 182.384 213.600 183.392 213.600 C 184.202 213.600,184.400 213.798,184.400 214.604 C 184.400 215.830,186.381 218.000,187.500 218.000 C 188.034 218.000,188.399 218.343,188.564 219.000 L 188.815 220.000 197.008 220.000 L 205.200 220.000 205.200 219.000 C 205.200 218.251,205.411 218.000,206.040 218.000 C 206.502 218.000,207.492 217.388,208.240 216.640 C 208.988 215.892,209.600 214.902,209.600 214.440 C 209.600 213.811,209.851 213.600,210.600 213.600 L 211.600 213.600 211.600 205.408 L 211.600 197.215 210.600 196.964 C 209.923 196.794,209.600 196.436,209.599 195.857 C 209.598 195.351,209.107 194.672,208.400 194.200 C 207.741 193.760,207.202 193.265,207.201 193.100 C 207.201 192.935,206.750 192.800,206.200 192.800 C 205.400 192.800,205.200 192.600,205.200 191.800 L 205.200 190.800 197.067 190.800 C 192.593 190.800,188.813 190.920,188.667 191.067 M205.467 231.067 C 205.320 231.213,205.200 231.653,205.200 232.044 C 205.200 232.593,204.816 232.782,203.500 232.877 C 202.198 232.971,201.709 233.207,201.412 233.884 C 201.199 234.370,200.659 234.820,200.212 234.884 C 199.534 234.981,199.380 235.342,199.280 237.075 C 199.169 238.997,199.080 239.160,198.080 239.275 L 197.000 239.400 197.000 247.400 L 197.000 255.400 198.080 255.525 C 199.080 255.640,199.169 255.803,199.280 257.725 C 199.383 259.510,199.522 259.817,200.275 259.925 C 200.821 260.002,201.198 260.379,201.275 260.925 C 201.380 261.662,201.696 261.819,203.275 261.921 C 204.979 262.031,205.162 262.141,205.275 263.121 L 205.400 264.200 213.600 264.200 L 221.800 264.200 221.927 263.100 C 222.045 262.069,222.175 262.000,223.983 262.000 C 225.651 262.000,225.944 261.878,226.140 261.100 C 226.265 260.605,226.779 260.020,227.283 259.800 C 228.023 259.477,228.223 259.034,228.322 257.500 C 228.432 255.785,228.539 255.600,229.422 255.600 L 230.400 255.600 230.400 247.400 L 230.400 239.200 229.422 239.200 C 228.540 239.200,228.432 239.015,228.322 237.310 C 228.222 235.765,228.020 235.330,227.217 234.931 C 226.676 234.663,226.097 234.074,225.930 233.622 C 225.684 232.958,225.276 232.800,223.813 232.800 C 222.146 232.800,222.000 232.720,222.000 231.800 L 222.000 230.800 213.867 230.800 C 209.393 230.800,205.613 230.920,205.467 231.067 M164.254 261.113 C 164.143 261.292,164.196 261.670,164.371 261.953 C 164.619 262.354,164.741 262.331,164.927 261.846 C 165.179 261.190,164.598 260.557,164.254 261.113 M186.508 261.620 C 186.448 262.419,186.979 262.681,187.372 262.045 C 187.493 261.850,187.361 261.499,187.079 261.266 C 186.694 260.946,186.553 261.035,186.508 261.620 "
+                  stroke="none"
+                  fill="#cc0434"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path3"
+                  d="M123.477 131.856 C 123.325 132.009,123.200 132.553,123.200 133.067 C 123.200 133.798,122.981 134.000,122.190 134.000 L 121.179 134.000 121.290 140.100 C 121.397 146.031,121.424 146.203,122.275 146.325 C 122.821 146.402,123.198 146.779,123.275 147.325 C 123.397 148.176,123.569 148.203,129.500 148.310 L 135.600 148.421 135.600 147.437 C 135.600 146.661,135.832 146.426,136.700 146.327 L 137.800 146.200 137.800 140.200 L 137.800 134.200 136.724 134.076 C 135.893 133.980,135.620 133.707,135.524 132.876 L 135.400 131.800 129.577 131.690 C 126.374 131.629,123.629 131.704,123.477 131.856 M108.572 174.968 C 108.421 175.571,107.990 176.002,107.462 176.077 C 106.075 176.275,106.112 190.131,107.500 190.328 C 107.995 190.398,108.400 190.700,108.400 190.999 C 108.400 192.534,109.470 192.800,115.655 192.800 C 121.866 192.800,123.200 192.520,123.200 191.214 C 123.200 190.732,123.528 190.409,124.100 190.328 C 125.499 190.129,125.523 176.274,124.125 176.075 C 123.579 175.998,123.202 175.621,123.125 175.075 C 123.002 174.217,122.860 174.198,115.910 174.091 L 108.820 173.982 108.572 174.968 M102.400 216.973 C 102.400 217.738,102.164 217.974,101.300 218.073 C 100.523 218.163,100.163 218.457,100.075 219.075 C 99.998 219.621,99.621 219.998,99.075 220.075 C 98.212 220.198,98.200 220.298,98.200 227.400 C 98.200 234.502,98.212 234.602,99.075 234.725 C 99.621 234.802,99.998 235.179,100.075 235.725 C 100.163 236.343,100.523 236.637,101.300 236.727 C 102.164 236.826,102.400 237.062,102.400 237.827 L 102.400 238.800 109.600 238.800 L 116.800 238.800 116.800 237.800 C 116.800 237.000,117.000 236.800,117.800 236.800 C 118.593 236.800,118.800 236.598,118.800 235.827 C 118.800 235.062,119.036 234.826,119.900 234.727 L 121.000 234.600 121.110 227.853 C 121.224 220.835,121.058 220.000,119.551 220.000 C 119.017 220.000,118.800 219.712,118.800 219.000 C 118.800 218.200,118.600 218.000,117.800 218.000 C 117.000 218.000,116.800 217.800,116.800 217.000 L 116.800 216.000 109.600 216.000 L 102.400 216.000 102.400 216.973 M113.280 258.480 C 113.016 258.744,112.800 259.181,112.800 259.452 C 112.800 259.723,112.395 260.002,111.900 260.072 C 111.256 260.164,110.964 260.512,110.873 261.300 C 110.783 262.082,110.521 262.400,109.964 262.400 C 108.707 262.400,108.400 264.012,108.400 270.600 C 108.400 277.188,108.707 278.800,109.964 278.800 C 110.521 278.800,110.783 279.118,110.873 279.900 C 110.959 280.647,111.260 281.037,111.812 281.116 C 112.259 281.180,112.799 281.630,113.012 282.116 C 113.390 282.978,113.590 283.003,121.107 283.109 L 128.813 283.217 129.185 282.239 C 129.402 281.670,129.942 281.207,130.478 281.131 C 131.140 281.037,131.435 280.696,131.524 279.925 C 131.611 279.172,131.911 278.812,132.524 278.725 C 133.905 278.528,133.905 262.672,132.524 262.475 C 131.911 262.388,131.611 262.028,131.524 261.275 C 131.435 260.504,131.140 260.163,130.478 260.069 C 129.945 259.994,129.402 259.531,129.188 258.969 L 128.820 258.000 121.290 258.000 C 115.967 258.000,113.619 258.141,113.280 258.480 M140.276 291.858 C 140.124 292.009,140.000 292.543,140.000 293.045 C 140.000 293.848,139.775 293.971,138.100 294.078 L 136.200 294.200 136.078 296.100 C 135.968 297.815,135.861 298.000,134.978 298.000 L 134.000 298.000 134.000 305.400 L 134.000 312.800 134.978 312.800 C 135.861 312.800,135.968 312.985,136.078 314.700 L 136.200 316.600 138.100 316.722 C 139.815 316.832,140.000 316.939,140.000 317.822 L 140.000 318.800 147.400 318.800 L 154.800 318.800 154.800 317.822 C 154.800 316.939,154.985 316.832,156.700 316.722 L 158.600 316.600 158.721 314.725 C 158.831 313.021,158.941 312.838,159.921 312.725 L 161.000 312.600 161.000 305.400 L 161.000 298.200 159.921 298.075 C 158.941 297.962,158.831 297.779,158.721 296.075 L 158.600 294.200 156.725 294.079 C 155.021 293.969,154.838 293.859,154.725 292.879 L 154.600 291.800 147.576 291.691 C 143.712 291.631,140.427 291.706,140.276 291.858 M230.800 313.778 C 230.800 314.661,230.615 314.768,228.900 314.878 C 227.396 314.975,226.920 315.182,226.616 315.874 C 226.405 316.355,225.820 316.936,225.316 317.165 C 224.558 317.511,224.400 317.894,224.400 319.391 C 224.400 321.058,224.321 321.200,223.392 321.200 L 222.384 321.200 222.492 329.500 C 222.599 337.706,222.610 337.801,223.479 337.925 C 224.221 338.031,224.377 338.344,224.479 339.925 C 224.582 341.521,224.734 341.819,225.500 341.928 C 226.083 342.011,226.400 342.331,226.400 342.838 C 226.400 343.715,227.803 344.400,229.598 344.400 C 230.482 344.400,230.774 344.606,230.872 345.300 C 231.000 346.199,231.008 346.200,239.000 346.200 C 246.992 346.200,247.000 346.199,247.128 345.300 C 247.226 344.606,247.518 344.400,248.402 344.400 C 250.197 344.400,251.600 343.715,251.600 342.838 C 251.600 342.331,251.917 342.011,252.500 341.928 C 253.266 341.819,253.418 341.521,253.521 339.925 C 253.623 338.344,253.779 338.031,254.521 337.925 C 255.390 337.801,255.401 337.706,255.508 329.500 L 255.616 321.200 254.608 321.200 C 253.668 321.200,253.600 321.068,253.600 319.243 C 253.600 317.548,253.480 317.256,252.700 317.060 C 252.205 316.935,251.620 316.421,251.400 315.917 C 251.077 315.177,250.634 314.977,249.100 314.878 C 247.385 314.768,247.200 314.661,247.200 313.778 L 247.200 312.800 239.000 312.800 L 230.800 312.800 230.800 313.778 M178.500 317.065 C 178.225 317.176,178.000 317.702,178.000 318.233 C 178.000 319.002,177.792 319.200,176.982 319.200 C 175.671 319.200,174.000 320.871,174.000 322.182 C 174.000 322.896,173.785 323.200,173.280 323.200 C 171.745 323.200,171.600 323.910,171.600 331.447 C 171.600 339.546,171.677 340.000,173.043 340.000 C 173.743 340.000,174.000 340.221,174.000 340.826 C 174.000 342.613,175.525 344.365,177.100 344.387 C 177.712 344.396,178.000 344.642,178.000 345.157 C 178.000 346.300,178.674 346.400,186.400 346.400 C 194.126 346.400,194.800 346.300,194.800 345.157 C 194.800 344.660,195.082 344.400,195.623 344.400 C 196.691 344.400,198.800 342.084,198.800 340.911 C 198.800 340.217,199.028 340.000,199.757 340.000 C 201.123 340.000,201.200 339.546,201.200 331.447 C 201.200 323.910,201.055 323.200,199.520 323.200 C 199.045 323.200,198.800 322.901,198.800 322.322 C 198.800 321.149,196.881 319.200,195.725 319.200 C 195.016 319.200,194.800 318.976,194.800 318.243 C 194.800 317.716,194.515 317.176,194.167 317.043 C 193.414 316.754,179.221 316.774,178.500 317.065 "
+                  stroke="none"
+                  fill="#0464fc"
+                  fill-rule="evenodd"
+                ></path>
+              </g>
+            </svg>
+            <div class="flex-grow"></div>
+            <a
+              class="card-nav"
+              href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#how-to"
+              target="_blank"
+            >
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Extending the benchmark
+            </a>
+          </div>
+        </div>
+      </section>
+      <section>
+        <div class="two-col">
+          <div class="flat-card">
+            <h3>📄 Paper</h3>
+            <div>
+              A preprint of our most recent paper is available on
+              <a href="https://arxiv.org/abs/2207.12560" target="_blank"
+                >ArXiv</a
+              >. It includes an in-depth discussion of the different design
+              decisions and its limiations, as well as a multi-faceted analysis
+              of results from large scale comparison across 9 frameworks on more
+              than 100 tasks.
+              <!-- https://arxiv.org/abs/1907.00909 -->
+            </div>
+          </div>
+          <div class="flat-card">
+            <h3>🧑‍💻 Code</h3>
+            <div>
+              The entire benchmark tool is open source and developed on
+              <a
+                href="https://github.com/openml/automlbenchmark"
+                target="_blank"
+                >Github</a
+              >. The Github discussion board and issue trackers are the main way
+              for us to interact with the community.
+            </div>
+          </div>
+        </div>
+        <!-- <h2>Paper <img src=/></h2> -->
+      </section>
+      <section>
+        <h2>AutoML Frameworks</h2>
+        <div>
+          <div class="grey">
+            Many AutoML frameworks are already integrated with the AutoML
+            benchmark tool and
+            <a href="#ADDFRAMEWORK">adding more is easy</a>. We have more
+            information about the different frameworks on our
+            <a href="#FRAMEWORKS">framework overview page</a>. The icons below
+            link directly to their respective Github repositories.
+          </div>
+          <div class="icon-flex">
+            <a
+              href="https://github.com/openml-labs/gama"
+              target="_blank"
+              class="framework-logo"
+              ><img src="img/logos/GAMA-icon.png" title="GAMA"
+            /></a>
+            <a href="https://github.com/EpistasisLab/tpot" target="_blank"
+              ><img src="img/logos/tpot.jpeg" title="TPOT"
+            /></a>
+            <a href="https://github.com/awslabs/autogluon" target="_blank"
+              ><img src="img/logos/autogluon.png" title="AutoGluon"
+            /></a>
+            <a
+              href="https://github.com/automl/auto-sklearn"
+              target="_blank"
+              class="framework-logo"
+              ><img src="img/logos/auto-sklearn.png" title="auto-sklearn"
+            /></a>
+            <a href="https://github.com/mljar/mljar-supervised" target="_blank"
+              ><img src="img/logos/mljar.png" title="mljar-supervised"
+            /></a>
+            <a href="https://github.com/h2oai/h2o-3" target="_blank"
+              ><img src="img/logos/h2o-automl-logo.jpeg" title="H2O AutoML"
+            /></a>
+            <a
+              href="https://github.com/sb-ai-lab/LightAutoML"
+              target="_blank"
+              class="framework-logo"
+              ><img
+                src="img/logos/LightAutoML_logo_small.png"
+                title="Light AutoML"
+            /></a>
+            <a href="https://github.com/microsoft/FLAML" target="_blank"
+              ><img src="img/logos/flaml.svg" title="FLAML"
+            /></a>
+          </div>
+          <div></div>
+        </div>
+      </section>
+      <section>
+        <h2>Community</h2>
+        <div class="grey">
+          We welcome any contributions to the AutoML benchmark. Our goal is to
+          provide the best benchmark tools for AutoML research and we can't do
+          that on our own. Contributions are appreciated in many forms,
+          including feedback on the benchmark design, feature requests, bug
+          reports, code and documentation contributions, and more. Why not stop
+          by on our
+          <a
+            href="https://github.com/openml/automlbenchmark/discussions/216"
+            target="_blank"
+            >welcome board</a
+          >
+          and let us know what got you interested in the benchmark?
+        </div>
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/papers.html b/docs/papers.html
new file mode 100644
index 000000000..3d25ceaaa
--- /dev/null
+++ b/docs/papers.html
@@ -0,0 +1,308 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Papers</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <h1 class="page-title">Papers</h1>
+      <div class="paper">
+        <h2 class="paper-title">AMLB: an AutoML Benchmark</h2>
+        <div class="paper-authors">
+          Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell,
+          Sébastien Poirier, Janek Thomas, Bernd Bischl and Joaquin Vanschoren
+        </div>
+
+        <!-- <div class="paper-year">2019</div>
+      <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+      <div class="paper-venue">cite</div> -->
+
+        <div class="paper-abstract">
+          Comparing different AutoML frameworks is notoriously challenging and
+          often done incorrectly. We introduce an open and extensible benchmark
+          that follows best practices and avoids common mistakes when comparing
+          AutoML frameworks. We conduct a thorough comparison of 9 well-known
+          AutoML frameworks across 71 classification and 33 regression tasks.
+          The differences between the AutoML frameworks are explored with a
+          multi-faceted analysis, evaluating model accuracy, its trade-offs with
+          inference time, and framework failures. We also use Bradley-Terry
+          trees to discover subsets of tasks where the relative AutoML framework
+          rankings differ. The benchmark comes with an open-source tool that
+          integrates with many AutoML frameworks and automates the empirical
+          evaluation process end-to-end: from framework installation and
+          resource allocation to in-depth evaluation. The benchmark uses public
+          data sets, can be easily extended with other AutoML frameworks and
+          tasks, and has a website with up-to-date results.
+        </div>
+        <div class="paper-links">
+          <div class="hover-expand">
+            <strong>2022</strong>
+            <div>Under Review</div>
+          </div>
+          <a class="flat-button">
+            <label for="cite-2022">CITE</label>
+          </a>
+          <a href="https://arxiv.org/pdf/2207.12560.pdf" target="_blank">PDF</a>
+          <a href="https://arxiv.org/abs/2207.12560" target="_blank">ArXiv</a>
+        </div>
+        <div>
+          <input
+            type="checkbox"
+            id="cite-2022"
+            class="hidden-content-checkbox"
+          />
+          <div class="hidden-content bibtex">
+            @misc{https://doi.org/10.48550/arxiv.2207.12560, <br />
+            &emsp; doi = {10.48550/ARXIV.2207.12560}, <br />
+            &emsp; url = {https://arxiv.org/abs/2207.12560}, <br />
+            &emsp; author = {Gijsbers, Pieter and Bueno, Marcos L. P. and Coors,
+            Stefan and LeDell, Erin and Poirier, S\'{e}bastien and Thomas, Janek
+            and Bischl, Bernd and Vanschoren, Joaquin}, <br />
+            &emsp; keywords = {Machine Learning (cs.LG), Machine Learning
+            (stat.ML), FOS: Computer and information sciences, FOS: Computer and
+            information sciences}, <br />
+            &emsp; title = {AMLB: an AutoML Benchmark}, <br />
+            &emsp; publisher = {arXiv}, <br />
+            &emsp; year = {2022}, <br />
+            &emsp; copyright = {Creative Commons Attribution 4.0 International}
+            <br />
+            }
+          </div>
+        </div>
+      </div>
+      <div class="paper">
+        <h2 class="paper-title">An Open Source AutoML Benchmark</h2>
+        <div class="paper-authors">
+          Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd
+          Bischl, Joaquin Vanschoren
+        </div>
+
+        <!-- <div class="paper-year">2019</div>
+      <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+      <div class="paper-venue">cite</div> -->
+
+        <div class="paper-abstract">
+          In recent years, an active field of research has developed around
+          automated machine learning (AutoML). Unfortunately, comparing
+          different AutoML systems is hard and often done incorrectly. We
+          introduce an open, ongoing, and extensible benchmark framework which
+          follows best practices and avoids common mistakes. The framework is
+          open-source, uses public datasets and has a website with up-to-date
+          results. We use the framework to conduct a thorough comparison of 4
+          AutoML systems across 39 datasets and analyze the results.
+        </div>
+        <div class="paper-links">
+          <div class="hover-expand">
+            <strong>2019</strong>
+            <div>ICML 2019 AutoML Workshop</div>
+          </div>
+          <a class="flat-button">
+            <label for="cite-2019">CITE</label>
+          </a>
+          <a
+            href="https://www.automl.org/wp-content/uploads/2019/06/automlws2019_Paper45.pdf"
+            target="_blank"
+            >PDF</a
+          >
+          <a href="https://arxiv.org/abs/1907.00909" target="_blank">ArXiv</a>
+        </div>
+        <div>
+          <input
+            type="checkbox"
+            id="cite-2019"
+            class="hidden-content-checkbox"
+          />
+          <div class="hidden-content bibtex">
+            @article{amlb2019, <br />
+            &emsp; title={An Open Source AutoML Benchmark},<br />
+            &emsp; author={Gijsbers, P. and LeDell, E. and Poirier, S. and
+            Thomas, J. and Bischl, B. and Vanschoren, J.}, <br />
+            &emsp; journal={arXiv preprint arXiv:1907.00909 [cs.LG]}, <br />
+            &emsp; url={https://arxiv.org/abs/1907.00909}, <br />
+            &emsp; note={Accepted at AutoML Workshop at ICML 2019}, <br />
+            &emsp; year={2019} <br />}
+          </div>
+        </div>
+      </div>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/results.html b/docs/results.html
new file mode 100644
index 000000000..504ee0f07
--- /dev/null
+++ b/docs/results.html
@@ -0,0 +1,281 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Results</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <h1 class="page-title">Results</h1>
+        <div class="accordion acard">
+          <div>
+            ⚠️ <a href="papers.html">Our paper</a> outlines
+            <strong>important limitations</strong> for the interpretation of
+            results. These limitations include:
+          </div>
+          <input
+            type="checkbox"
+            id="more-limitations"
+            class="accordion-input"
+          />
+          <div class="accordion-content">
+            <ul class="limitations-list">
+              <li>
+                We use AutoML framework versions from
+                <strong>September 2021</strong>, many frameworks have since seen
+                major updates.
+              </li>
+              <li>
+                We use the "benchmark" modes of the frameworks, which generally
+                <em>only</em> optimize for performance. Most AutoML frameworks
+                have multiple modes to support different use cases.
+              </li>
+              <li>
+                Results can not be used to make conclusions about which
+                algorithm is best, as all frameworks differ in multiple ways.
+              </li>
+              <li>
+                Performance statistics are often independent from many
+                qualitative differences, such as ease of use or
+                interpretability.
+              </li>
+            </ul>
+            Please read Section 5.3 in our paper for a more elaborate discussion
+            of these and other limitations.
+          </div>
+          <label for="more-limitations">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+      </section>
+      <section class="results">
+        <div>
+          All results are available as
+          <a href="https://test.openml.org/amlb/" target="_blank">raw files</a>
+          📂, but we open source the tools we used for generating the figures in
+          our paper. The best way to explore the results is through our
+          interactive
+          <a
+            href="https://compstat-lmu.shinyapps.io/AutoML-Benchmark-Analysis/"
+            target="_blank"
+            >Shiny app</a
+          >. It loads the latest results by default. It is also possible to use
+          our
+          <a
+            href="https://github.com/openml/automlbenchmark/tree/master/reports"
+            target="_blank"
+            >notebooks</a
+          >
+          which contain additional visualizations.
+        </div>
+        <img src="img/shiny.png" height="320px" class="shiny" />
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/style.css b/docs/style.css
new file mode 100644
index 000000000..73340f289
--- /dev/null
+++ b/docs/style.css
@@ -0,0 +1,527 @@
+* {
+  margin: 0;
+  padding: 0;
+  box-sizing: border-box;
+  font-size: 18px;
+}
+.grey {
+  color: #495057;
+}
+
+/* main: #1971c2 */
+
+body {
+  font-family: "lato", sans-serif;
+  color: #343a40;
+}
+
+section {
+  margin-bottom: 64px;
+}
+
+a {
+  color: 1971c2;
+}
+
+.page-content {
+  padding: 0 clamp(15%, 100%, calc((100vw - 1000px) / 2)); /* should maybe set width in px instead */
+}
+
+h1 {
+  font-size: 72px;
+  padding-bottom: 24px;
+}
+
+h2 {
+  font-size: 48px;
+  padding-bottom: 24px;
+}
+
+h3 {
+  font-size: 32px;
+  padding-bottom: 16px;
+}
+
+footer {
+  color: #e7f5ff;
+  font-size: 14px;
+  background-color: #1971c2;
+  text-align: center;
+  padding: 8px;
+}
+
+footer > a {
+  color: #e7f5ff;
+  font-size: 14px;
+}
+
+.terminal {
+  background-color: #343a40;
+  color: #f8f9fa;
+  font-family: "Inconsolata", sans-serif;
+  padding: 8px;
+  margin: 16px 5%;
+}
+
+/* NAVBAR */
+.construction-banner {
+  background-color: #1c7ed6;
+  color: white;
+  text-align: center;
+}
+
+.navigation-bar {
+  background-color: #1971c2;
+  color: white;
+  text-transform: uppercase;
+
+  display: flex;
+  gap: 24px;
+  justify-content: flex-end;
+  align-items: center;
+  padding: 16px;
+  height: 60px;
+}
+
+nav a:first-child {
+  flex-grow: 1;
+}
+
+.nav-icon {
+  height: 24px;
+  fill: white;
+}
+
+.nav-icon-with-text {
+  height: 18px;
+}
+
+.nav-icon:hover,
+.nav-icon:active {
+  height: 28px;
+}
+
+.nav-link:link,
+.nav-link:visited {
+  text-decoration: none;
+  color: white;
+}
+.nav-link:hover,
+.nav-link:active {
+  color: white;
+  font-weight: bold;
+}
+
+/* TITLE */
+
+.title,
+.subtitle {
+  color: #343a40;
+  text-align: center;
+}
+
+.title {
+  font-size: 74px;
+  padding-top: 48px;
+  font-weight: bold;
+}
+
+.subtitle {
+  font-size: 62px;
+  font-weight: bold;
+  padding-bottom: 24px;
+}
+
+.summary {
+  font-size: 24px;
+  color: #495057;
+  padding: 0 max((100% - 900px) / 2, 0px); /* does not seem to work with just the math expression */
+}
+
+/* CARDS */
+
+.three-cols {
+  display: grid;
+  grid-template-columns: repeat(3, 1fr);
+  column-gap: 5%;
+}
+
+.card {
+  box-shadow: 0 5px 10px rgba(0, 0, 0, 0.1);
+  border-radius: 5px;
+  padding: 16px;
+
+  display: flex;
+  flex-direction: column;
+}
+
+.card img {
+  padding: 16px 2%;
+}
+
+.card > svg {
+  display: block;
+  margin: 16px auto;
+}
+
+.flex-grow {
+  flex-grow: 1;
+}
+
+.card > .card-nav {
+  color: #1971c2;
+  text-decoration: none;
+
+  display: flex;
+  align-items: center;
+  justify-content: flex-start;
+  gap: 8px;
+}
+
+.card > .card-nav > svg {
+  height: 24px;
+  fill: #1971c2;
+}
+
+h3 {
+  font-size: 24px;
+  padding-bottom: 10px;
+}
+
+/* Frameworks */
+/* .icon-grid {
+  padding: 24px;
+  display: grid;
+  grid-auto-rows: 60px;
+} */
+.icon-flex {
+  margin-top: 24px;
+  display: flex;
+  flex-wrap: wrap;
+  justify-content: center;
+  align-items: center;
+  gap: 40px;
+}
+
+.icon-flex a img {
+  height: 90px;
+}
+/* Paper */
+
+.two-col {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  column-gap: 2%;
+
+  background-color: #1971c2;
+  color: #e7f5ff;
+  border-radius: 5px;
+}
+
+.two-col a {
+  color: white;
+}
+
+.two-col h3 {
+  color: white;
+}
+
+.flat-card {
+  padding: 16px;
+}
+
+/* FRAMEWORKS PAGE */
+.page-title {
+  margin-top: 72px;
+}
+
+.framework-card-list {
+  background-color: #e7f5ff;
+  padding: 32px 5%;
+  border-radius: 10px;
+}
+
+.framework-card-list > .accordion {
+  margin-bottom: 32px;
+}
+/* 
+.framework-card {
+  background-color: white;
+  margin-bottom: 64;
+  padding: 32;
+  border-radius: 10px;
+} */
+.framework-card:last-of-type {
+  margin-bottom: 0;
+}
+
+.framework-header {
+  display: flex;
+  align-items: center;
+  gap: 16px;
+  flex-wrap: wrap;
+}
+
+.framework-header > h3 {
+  padding-bottom: 0px;
+  font-size: 32px;
+}
+
+.framework-links {
+  display: flex;
+  gap: 16px;
+  justify-content: flex-end;
+
+  flex-grow: 1;
+}
+
+.framework-links > a {
+  text-decoration: none;
+  font-size: 24px;
+}
+
+/* RESULTS */
+
+.limitations-list {
+  padding: 0px 32px 16px 32px;
+}
+
+.limitations-list li {
+  padding: 8px 0px;
+}
+
+.results {
+  display: flex;
+  gap: 40px;
+}
+
+/* Accordions */
+.accordion {
+}
+
+.acard {
+  box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2);
+  border-radius: 10px;
+  background-color: white;
+}
+
+.accordion > div {
+  padding: 8px 32px;
+}
+
+.accordion > div:first-child {
+  padding-top: 16px;
+}
+
+.accordion > label {
+  width: 100%;
+  display: flex;
+  justify-content: center;
+}
+
+.accordion-input {
+  display: none;
+}
+
+.accordion-icon {
+  height: 24px;
+}
+
+.accordion-input:checked ~ .accordion-content {
+  display: block;
+}
+
+.accordion-input:checked ~ label .accordion-chevron-down {
+  display: none;
+}
+
+.accordion-input:checked ~ label .accordion-chevron-up {
+  display: block;
+}
+
+.accordion-input ~ .accordion-content {
+  display: none;
+}
+
+.accordion-input ~ label .accordion-chevron-up {
+  display: none;
+}
+
+.accordion-input ~ label .accordion-chevron-down {
+  display: block;
+}
+
+.accordion > label {
+  background: linear-gradient(0deg, rgba(0, 0, 0, 0.02), white);
+  border-radius: 10px;
+  padding-bottom: 8px;
+}
+
+.accordion > label:hover {
+  cursor: pointer;
+}
+
+/* Papers */
+.paper {
+  margin-bottom: 48px;
+}
+
+.paper-title {
+  padding-bottom: 0px;
+  margin-bottom: 0px;
+}
+
+.paper-links {
+  display: flex;
+  align-items: center;
+  gap: 12px;
+
+  margin-bottom: 16px;
+}
+
+.paper-links > a {
+  color: white;
+  background-color: #1c7ed6;
+  padding: 4px 12px;
+  border-radius: 4px;
+
+  text-decoration: none;
+  text-transform: uppercase;
+}
+
+.paper-year {
+  color: #343a40;
+  display: inline;
+  font-weight: bold;
+}
+
+.paper-venue {
+  color: #495057;
+  display: inline;
+}
+
+.paper-authors {
+  color: #4263eb;
+  padding-bottom: 8px;
+}
+
+.paper-abstract {
+  color: #495057;
+  padding-bottom: 16px;
+}
+
+.hover-expand {
+  display: flex;
+  gap: 20px;
+
+  color: #343a40;
+  background-color: white;
+  border: 2px solid #495057;
+  border-radius: 4px;
+  padding: 2px 8px;
+}
+
+.hover-expand > div {
+  display: none;
+}
+
+.hover-expand:hover > div {
+  display: block;
+}
+
+.hidden-content {
+  display: none;
+}
+
+.hidden-content-checkbox {
+  display: none;
+}
+
+.hidden-content-checkbox:checked ~ .hidden-content {
+  display: block;
+}
+
+.bibtex {
+  font-family: "Inconsolata", sans-serif;
+  color: #343a40;
+  background-color: #e7f5ff;
+  border-radius: 10px;
+  padding: 16px;
+}
+
+.flat-button {
+  cursor: pointer;
+}
+
+.flat-button > label {
+  cursor: pointer;
+}
+
+.mobile {
+  display: none;
+}
+
+.nav-mobile {
+  justify-content: space-between;
+  display: none;
+}
+
+@media (max-width: 900px) {
+  .desktop {
+    display: none;
+  }
+
+  .mobile {
+    display: block;
+    padding: 25px;
+  }
+
+  .shiny {
+    height: auto;
+    width: 100%;
+  }
+
+  .results {
+    flex-wrap: wrap;
+  }
+
+  .framework-header > h3 {
+    order: -1;
+    flex-basis: 100%;
+  }
+
+  h1 {
+    font-size: 48px;
+    padding-bottom: 18px;
+  }
+
+  .page-title {
+    margin-top: 32px;
+  }
+
+  .page-content {
+    padding: 0 32px;
+  }
+
+  .three-cols {
+    grid-template-columns: 1fr;
+    row-gap: 32px;
+  }
+
+  .two-col {
+    grid-template-columns: 1fr;
+    row-gap: 32px;
+  }
+
+  .nav-mobile {
+    display: flex;
+  }
+}
+
+.mobile * {
+  color: white;
+}
+
+.mobile > .summary > a {
+  font-size: 24px;
+}
diff --git a/frameworks/AutoGluon/README.md b/frameworks/AutoGluon/README.md
new file mode 100644
index 000000000..51286533e
--- /dev/null
+++ b/frameworks/AutoGluon/README.md
@@ -0,0 +1,16 @@
+# AutoGluon
+
+To run v0.5.2: ```python3 ../automlbenchmark/runbenchmark.py autogluon ...```
+
+To run mainline: ```python3 ../automlbenchmark/runbenchmark.py autogluonts:latest ...```
+
+
+# AutoGluonTS
+
+AutoGluonTS stands for autogluon.timeseries. This framework handles time series problems.
+
+## Run Steps
+
+To run v0.5.2: ```python3 ../automlbenchmark/runbenchmark.py autogluonts timeseries ...```
+
+To run mainline: ```python3 ../automlbenchmark/runbenchmark.py autogluonts:latest timeseries ...```
diff --git a/frameworks/AutoGluon/__init__.py b/frameworks/AutoGluon/__init__.py
index be2c15147..9d3d980a3 100644
--- a/frameworks/AutoGluon/__init__.py
+++ b/frameworks/AutoGluon/__init__.py
@@ -1,15 +1,23 @@
-from amlb.benchmark import TaskConfig
-from amlb.data import Dataset
+
 from amlb.utils import call_script_in_same_dir
+from amlb.benchmark import TaskConfig
+from amlb.data import Dataset, DatasetType
+from copy import deepcopy
 
 
 def setup(*args, **kwargs):
     call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
 
-
 def run(dataset: Dataset, config: TaskConfig):
-    from frameworks.shared.caller import run_in_venv
 
+    if dataset.type is not DatasetType.timeseries:
+        return run_autogluon_tabular(dataset, config)
+
+    else:
+        return run_autogluon_timeseries(dataset, config)
+
+def run_autogluon_tabular(dataset: Dataset, config: TaskConfig):
+    from frameworks.shared.caller import run_in_venv
     data = dict(
         train=dict(path=dataset.train.data_path('parquet')),
         test=dict(path=dataset.test.data_path('parquet')),
@@ -17,9 +25,38 @@ def run(dataset: Dataset, config: TaskConfig):
             name=dataset.target.name,
             classes=dataset.target.values
         ),
-        problem_type=dataset.type.name  # AutoGluon problem_type is using same names as amlb.data.DatasetType
+        problem_type=dataset.type.name,  # AutoGluon problem_type is using same names as amlb.data.DatasetType
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
 
+def run_autogluon_timeseries(dataset: Dataset, config: TaskConfig):
+    from frameworks.shared.caller import run_in_venv
+    dataset = deepcopy(dataset)
+    if not hasattr(dataset, 'timestamp_column'):
+        dataset.timestamp_column = None
+    if not hasattr(dataset, 'id_column'):
+        dataset.id_column = None
+    if not hasattr(dataset, 'forecast_range_in_steps'):
+        raise AttributeError("Unspecified `forecast_range_in_steps`.")
+
+    data = dict(
+        # train=dict(path=dataset.train.data_path('parquet')),
+        # test=dict(path=dataset.test.data_path('parquet')),
+        train=dict(path=dataset.train.path),
+        test=dict(path=dataset.test.path),
+        target=dict(
+            name=dataset.target.name,
+            classes=dataset.target.values
+        ),
+        problem_type=dataset.type.name,  # AutoGluon problem_type is using same names as amlb.data.DatasetType
+        timestamp_column=dataset.timestamp_column,
+        id_column=dataset.id_column,
+        forecast_range_in_steps=dataset.forecast_range_in_steps
+    )
+
+    return run_in_venv(__file__, "exec_ts.py",
+                       input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/AutoGluon/exec.py b/frameworks/AutoGluon/exec.py
index 7a7cfbb91..1884c7919 100644
--- a/frameworks/AutoGluon/exec.py
+++ b/frameworks/AutoGluon/exec.py
@@ -4,6 +4,8 @@
 import warnings
 import sys
 import tempfile
+from typing import Union
+
 warnings.simplefilter("ignore")
 
 if sys.platform == 'darwin':
@@ -13,12 +15,13 @@
 import pandas as pd
 matplotlib.use('agg')  # no need for tk
 
-from autogluon.tabular import TabularPredictor
+from autogluon.tabular import TabularPredictor, TabularDataset
 from autogluon.core.utils.savers import save_pd, save_pkl
 import autogluon.core.metrics as metrics
 from autogluon.tabular.version import __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, zip_path
 
 log = logging.getLogger(__name__)
@@ -46,7 +49,7 @@ def run(dataset, config):
     is_classification = config.type == 'classification'
     training_params = {k: v for k, v in config.framework_params.items() if not k.startswith('_')}
 
-    train, test = dataset.train.path, dataset.test.path
+    train_path, test_path = dataset.train.path, dataset.test.path
     label = dataset.target.name
     problem_type = dataset.problem_type
 
@@ -59,21 +62,35 @@ def run(dataset, config):
             path=models_dir,
             problem_type=problem_type,
         ).fit(
-            train_data=train,
+            train_data=train_path,
             time_limit=config.max_runtime_seconds,
             **training_params
         )
 
-    del train
+    # Persist model in memory that is going to be predicting to get correct inference latency
+    predictor.persist_models('best')
+
+    def inference_time_classification(data: Union[str, pd.DataFrame]):
+        return None, predictor.predict_proba(data, as_multiclass=True)
+
+    def inference_time_regression(data: Union[str, pd.DataFrame]):
+        return predictor.predict(data, as_pandas=False), None
 
+    infer = inference_time_classification if is_classification else inference_time_regression
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = pd.read_parquet(dataset.test.path)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+
+    test_data = TabularDataset(test_path)
+    with Timer() as predict:
+        predictions, probabilities = infer(test_data)
     if is_classification:
-        with Timer() as predict:
-            probabilities = predictor.predict_proba(test, as_multiclass=True)
         predictions = probabilities.idxmax(axis=1).to_numpy()
-    else:
-        with Timer() as predict:
-            predictions = predictor.predict(test, as_pandas=False)
-        probabilities = None
 
     prob_labels = probabilities.columns.values.astype(str).tolist() if probabilities is not None else None
 
@@ -82,7 +99,7 @@ def run(dataset, config):
     leaderboard_kwargs = dict(silent=True, extra_info=_leaderboard_extra_info)
     # Disabled leaderboard test data input by default to avoid long running computation, remove 7200s timeout limitation to re-enable
     if _leaderboard_test:
-        leaderboard_kwargs['data'] = test
+        leaderboard_kwargs['data'] = test_data
 
     leaderboard = predictor.leaderboard(**leaderboard_kwargs)
     with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', 1000):
@@ -105,7 +122,8 @@ def run(dataset, config):
                   models_count=num_models_trained,
                   models_ensemble_count=num_models_ensemble,
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,)
 
 
 def save_artifacts(predictor, leaderboard, config):
diff --git a/frameworks/AutoGluon/exec_ts.py b/frameworks/AutoGluon/exec_ts.py
new file mode 100644
index 000000000..ab7c4110f
--- /dev/null
+++ b/frameworks/AutoGluon/exec_ts.py
@@ -0,0 +1,172 @@
+import logging
+import os
+import shutil
+import warnings
+import sys
+import tempfile
+import numpy as np
+warnings.simplefilter("ignore")
+
+if sys.platform == 'darwin':
+    os.environ['OMP_NUM_THREADS'] = '1'
+
+import pandas as pd
+
+from autogluon.core.utils.savers import save_pd, save_pkl
+from autogluon.tabular import TabularDataset
+from autogluon.timeseries import TimeSeriesPredictor, TimeSeriesDataFrame
+from autogluon.timeseries.version import __version__
+
+from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.utils import Timer, zip_path
+
+log = logging.getLogger(__name__)
+
+
+def run(dataset, config):
+    log.info(f"\n**** AutoGluon TimeSeries [v{__version__}] ****\n")
+
+    timestamp_column = dataset.timestamp_column
+    id_column = dataset.id_column
+    prediction_length = dataset.forecast_range_in_steps
+
+    eval_metric = get_eval_metric(config)
+    label = dataset.target.name
+    time_limit = config.max_runtime_seconds
+
+    training_params = {k: v for k, v in config.framework_params.items() if not k.startswith('_')}
+
+    train_data, test_data = load_data(train_path=dataset.train.path,
+                                      test_path=dataset.test.path,
+                                      timestamp_column=timestamp_column,
+                                      id_column=id_column)
+    test_data_past = test_data.copy().slice_by_timestep(slice(None, -prediction_length))
+
+    predictor_path = tempfile.mkdtemp() + os.sep
+    with Timer() as training:
+        predictor = TimeSeriesPredictor(
+            target=label,
+            path=predictor_path,
+            prediction_length=prediction_length,
+            eval_metric=eval_metric,
+        )
+        predictor.fit(
+            train_data=train_data,
+            time_limit=time_limit,
+            **training_params,
+        )
+
+    with Timer() as predict:
+        predictions = predictor.predict(test_data_past)
+    log.info(predictions)
+
+    predictions_only = predictions['mean'].values
+    test_data_future = test_data.copy().slice_by_timestep(slice(-prediction_length, None))
+    truth_only = test_data_future[label].values
+
+    log.info(predictions_only)
+    log.info(truth_only)
+
+    leaderboard = predictor.leaderboard(test_data, silent=True)
+
+    with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', 1000):
+        log.info(leaderboard)
+
+    num_models_trained = len(leaderboard)
+
+    save_artifacts(predictor=predictor, leaderboard=leaderboard, config=config)
+    shutil.rmtree(predictor.path, ignore_errors=True)
+
+    quantiles = predictions.drop(columns=['mean']).reset_index(drop=True)
+    period_length = 1 # TODO: This period length could be adapted to the Dataset, but then we need to pass this information as well. As of now this works.
+
+    # we aim to calculate the mean period error from the past for each sequence: 1/N sum_{i=1}^N |x(t_i) - x(t_i - T)|
+    # 1. retrieve item_ids for each sequence/item
+    #dataset..X /. y
+    item_ids, inverse_item_ids = np.unique(test_data.reset_index()["item_id"].squeeze().to_numpy(), return_index=False, return_inverse=True)
+    # 2. capture sequences in a list
+    y_past = [test_data[label].squeeze().to_numpy()[inverse_item_ids == i][:-prediction_length] for i in range(len(item_ids))]
+    # 3. calculate period error per sequence
+    y_past_period_error = [np.abs(y_past_item[period_length:] - y_past_item[:-period_length]).mean() for y_past_item in y_past]
+    # 4. repeat period error for each sequence, to save one for each element
+    y_past_period_error_rep = np.repeat(y_past_period_error, prediction_length)
+
+    optional_columns = quantiles
+    optional_columns = optional_columns.assign(y_past_period_error=y_past_period_error_rep)
+
+    return result(output_file=config.output_predictions_file,
+                  predictions=predictions_only,
+                  truth=truth_only,
+                  probabilities=None,
+                  probabilities_labels=None,
+                  target_is_encoded=False,
+                  models_count=num_models_trained,
+                  training_duration=training.duration,
+                  predict_duration=predict.duration,
+                  optional_columns=optional_columns)
+
+def load_data(train_path, test_path, timestamp_column, id_column):
+
+    train_df = pd.read_csv(
+        train_path,
+        parse_dates=[timestamp_column],
+    )
+
+    train_data = TimeSeriesDataFrame.from_data_frame(
+        train_df,
+        id_column=id_column,
+        timestamp_column=timestamp_column,
+    )
+
+    test_df = pd.read_csv(
+        test_path,
+        parse_dates=[timestamp_column],
+    )
+
+    test_data = TimeSeriesDataFrame.from_data_frame(
+        test_df,
+        id_column=id_column,
+        timestamp_column=timestamp_column,
+    )
+
+    return train_data, test_data
+
+
+def get_eval_metric(config):
+    # TODO: Support more metrics
+    metrics_mapping = dict(
+        mape="MAPE",
+        smape="sMAPE",
+        mase="MASE",
+        mse="MSE",
+        rmse="RMSE",
+    )
+
+    eval_metric = metrics_mapping[config.metric] if config.metric in metrics_mapping else None
+    if eval_metric is None:
+        log.warning("Performance metric %s not supported.", config.metric)
+    return eval_metric
+
+
+def save_artifacts(predictor, leaderboard, config):
+    artifacts = config.framework_params.get('_save_artifacts', ['leaderboard'])
+    try:
+        if 'leaderboard' in artifacts:
+            leaderboard_dir = output_subdir("leaderboard", config)
+            save_pd.save(path=os.path.join(leaderboard_dir, "leaderboard.csv"), df=leaderboard)
+
+        if 'info' in artifacts:
+            ag_info = predictor.info()
+            info_dir = output_subdir("info", config)
+            save_pkl.save(path=os.path.join(info_dir, "info.pkl"), object=ag_info)
+
+        if 'models' in artifacts:
+            shutil.rmtree(os.path.join(predictor.path, "utils"), ignore_errors=True)
+            models_dir = output_subdir("models", config)
+            zip_path(predictor.path, os.path.join(models_dir, "models.zip"))
+    except Exception:
+        log.warning("Error when saving artifacts.", exc_info=True)
+
+
+if __name__ == '__main__':
+    call_run(run)
diff --git a/frameworks/AutoGluon/setup.sh b/frameworks/AutoGluon/setup.sh
index 6e9a2a76f..967c9d40c 100755
--- a/frameworks/AutoGluon/setup.sh
+++ b/frameworks/AutoGluon/setup.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+
 HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/awslabs/autogluon.git"}
@@ -9,34 +10,37 @@ fi
 
 # creating local venv
 . ${HERE}/../shared/setup.sh ${HERE} true
-#if [[ -x "$(command -v apt-get)" ]]; then
-#    SUDO apt-get install -y libomp-dev
+
+# Below fixes seg fault on MacOS due to bug in libomp: https://github.com/awslabs/autogluon/issues/1442
 if [[ -x "$(command -v brew)" ]]; then
-    brew install libomp
+    wget https://raw.githubusercontent.com/Homebrew/homebrew-core/fb8323f2b170bd4ae97e1bac9bf3e2983af3fdb0/Formula/libomp.rb -P "${HERE}/lib"
+    brew install "${HERE}/lib/libomp.rb"
 fi
 
 PIP install --upgrade pip
 PIP install --upgrade setuptools wheel
-PIP install "mxnet<2.0.0"
-PIP install "scikit-learn-intelex<2021.3"
 
 if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
+    PIP install --no-cache-dir -U "${PKG}"
+    PIP install --no-cache-dir -U "${PKG}.tabular[skex]"
 elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+    PIP install --no-cache-dir -U "${PKG}==${VERSION}"
+    PIP install --no-cache-dir -U "${PKG}.tabular[skex]==${VERSION}"
 else
     TARGET_DIR="${HERE}/lib/${PKG}"
     rm -Rf ${TARGET_DIR}
     git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
     cd ${TARGET_DIR}
-    PIP install -e core/
-    PIP install -e features/
-    PIP install -e tabular/[all]
-    PIP install -e mxnet/
-    PIP install -e extra/
-    PIP install -e text/
-    PIP install -e vision/
-    PIP install -e autogluon/
+    PY_EXEC_NO_ARGS="$(cut -d' ' -f1 <<<"$py_exec")"
+    PY_EXEC_DIR=$(dirname "$PY_EXEC_NO_ARGS")
+    env PATH="$PY_EXEC_DIR:$PATH" bash -c ./full_install.sh
+    PIP install -e tabular/[skex]
 fi
 
-PY -c "from autogluon.tabular.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+if [[ ${MODULE} == "timeseries" ]]; then
+    PY -c "from autogluon.timeseries.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+    # TODO: GPU version install
+    PIP install "mxnet<2.0"
+else
+    PY -c "from autogluon.tabular.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+fi
diff --git a/frameworks/GAMA/__init__.py b/frameworks/GAMA/__init__.py
index f660e2f8f..750f5e74e 100644
--- a/frameworks/GAMA/__init__.py
+++ b/frameworks/GAMA/__init__.py
@@ -23,6 +23,8 @@ def run(dataset: Dataset, config: TaskConfig):
             y=dataset.test.y
         ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/GAMA/exec.py b/frameworks/GAMA/exec.py
index e0880bf34..d9e89a8e8 100644
--- a/frameworks/GAMA/exec.py
+++ b/frameworks/GAMA/exec.py
@@ -2,6 +2,9 @@
 import os
 import sys
 import tempfile as tmp
+from typing import Union
+
+import pandas as pd
 
 if sys.platform == 'darwin':
     os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'
@@ -18,7 +21,8 @@
 from gama.data_loading import file_to_pandas
 from gama import GamaClassifier, GamaRegressor, __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, touch
 
 
@@ -72,23 +76,29 @@ def run(dataset, config):
     gama_automl = estimator(**kwargs)
 
     X_train, y_train = dataset.train.X, dataset.train.y
-    # data = file_to_pandas(dataset.train.path, encoding='utf-8')
-    # X_train, y_train = data.loc[:, data.columns != dataset.target], data.loc[:, dataset.target]
-
     with Timer() as training_timer:
         gama_automl.fit(X_train, y_train)
 
     log.info('Predicting on the test set.')
-    X_test, y_test = dataset.test.X, dataset.test.y
-    # data = file_to_pandas(dataset.test.path, encoding='utf-8')
-    # X_test, y_test = data.loc[:, data.columns != dataset.target], data.loc[:, dataset.target]
-
+    def infer(data: Union[str, pd.DataFrame]):
+        test_data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = gama_automl.predict_proba if is_classification else gama_automl.predict
+        return predict_fn(test_data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
     with Timer() as predict_timer:
+        X_test, y_test = dataset.test.X, dataset.test.y
         predictions = gama_automl.predict(X_test)
+
+    probabilities = None
     if is_classification:
         probabilities = gama_automl.predict_proba(X_test)
-    else:
-        probabilities = None
 
     return result(
         output_file=config.output_predictions_file,
@@ -98,7 +108,8 @@ def run(dataset, config):
         target_is_encoded=False,
         models_count=len(gama_automl._final_pop),
         training_duration=training_timer.duration,
-        predict_duration=predict_timer.duration
+        predict_duration=predict_timer.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/H2OAutoML/__init__.py b/frameworks/H2OAutoML/__init__.py
index f8faf87a5..596513181 100644
--- a/frameworks/H2OAutoML/__init__.py
+++ b/frameworks/H2OAutoML/__init__.py
@@ -8,24 +8,17 @@ def setup(*args, **kwargs):
     call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
 
 
-# def version():
-#     from frameworks.shared.caller import run_cmd_in_venv
-#     out, err = run_cmd_in_venv(__file__, """{py} -c "from h2o import __version__; print(__version__)" | grep "^\d\." """)
-#     if err:
-#         raise ValueError(err)
-#     return out
-
-
 def run(dataset: Dataset, config: TaskConfig):
     from frameworks.shared.caller import run_in_venv
-
     data = dict(
         train=dict(path=dataset.train.path),
         test=dict(path=dataset.test.path),
         target=dict(index=dataset.target.index),
-        domains=dict(cardinalities=[0 if f.values is None else len(f.values) for f in dataset.features])
+        domains=dict(cardinalities=[0 if f.values is None else len(f.values) for f in dataset.features]),
+        format=dataset.train.format,
     )
-
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt=dataset.train.format, with_labels=True)
     config.ext.monitoring = rconfig().monitoring
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/H2OAutoML/exec.py b/frameworks/H2OAutoML/exec.py
index de215b172..d0288903a 100644
--- a/frameworks/H2OAutoML/exec.py
+++ b/frameworks/H2OAutoML/exec.py
@@ -1,6 +1,8 @@
 import contextlib
 import logging
 import os
+import pathlib
+
 import psutil
 import re
 
@@ -10,7 +12,8 @@
 import h2o
 from h2o.automl import H2OAutoML
 
-from frameworks.shared.callee import FrameworkError, call_run, output_subdir, result
+from frameworks.shared.callee import FrameworkError, call_run, output_subdir, result, \
+    measure_inference_times
 from frameworks.shared.utils import Monitoring, Namespace as ns, Timer, clean_dir, touch, zip_path
 
 log = logging.getLogger(__name__)
@@ -88,6 +91,10 @@ def run(dataset, config):
         test = h2o.import_file(dataset.test.path, destination_frame=frame_name('test', config), **import_kwargs)
         # test.impute(method='mean')
 
+        if config.type == 'classification' and dataset.format == 'csv':
+            train[dataset.target.index] = train[dataset.target.index].asfactor()
+            test[dataset.target.index] = test[dataset.target.index].asfactor()
+
         log.info("Running model on task %s, fold %s.", config.name, config.fold)
         log.debug("Running H2O AutoML with a maximum time of %ss on %s core(s), optimizing %s.",
                   config.max_runtime_seconds, config.cores, sort_metric)
@@ -101,8 +108,8 @@ def run(dataset, config):
                                            check_on_exit=True,
                                            verbosity=config.ext.monitoring.verbosity)
                    if config.framework_params.get('_monitor_backend', False)
-                   # else contextlib.nullcontext  # Py 3.7+ only
-                   else contextlib.contextmanager(lambda: (_ for _ in (0,)))()
+                   else contextlib.nullcontext()  # Py 3.7+ only
+                   # else contextlib.contextmanager(lambda: (_ for _ in (0,)))()
                    )
         with Timer() as training:
             with monitor:
@@ -111,6 +118,17 @@ def run(dataset, config):
         if not aml.leader:
             raise FrameworkError("H2O could not produce any model in the requested time.")
 
+        def infer(path: str):
+            filename = pathlib.Path(path).name
+            # H2O can't do inference on single row arff, it needs columns explicitly:
+            # https://github.com/h2oai/h2o-3/issues/15572
+            batch = h2o.import_file(path, col_names=train.names, destination_frame=frame_name(filename, config), **import_kwargs)
+            return aml.predict(batch)
+
+        inference_times = {}
+        if config.measure_inference_time:
+            inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+
         with Timer() as predict:
             preds = aml.predict(test)
 
@@ -125,7 +143,8 @@ def run(dataset, config):
             probabilities_labels=preds.probabilities_labels,
             models_count=len(aml.leaderboard),
             training_duration=training.duration,
-            predict_duration=predict.duration
+            predict_duration=predict.duration,
+            inference_times=inference_times,
         )
 
     finally:
diff --git a/frameworks/MLPlan/setup.sh b/frameworks/MLPlan/setup.sh
index f80e0d53c..5359f4cb2 100755
--- a/frameworks/MLPlan/setup.sh
+++ b/frameworks/MLPlan/setup.sh
@@ -26,7 +26,7 @@ rm -Rf ${TARGET_DIR}
 
 mkdir -p $DOWNLOAD_DIR
 echo "Download ML-Plan from extern"
-wget -q https://download.mlplan.org/version/$VERSION -O $DOWNLOAD_DIR/$MLPLAN_ARC
+wget -q --no-check-certificate https://download.mlplan.org/version/$VERSION -O $DOWNLOAD_DIR/$MLPLAN_ARC
 echo "Download finished. Now unzip the downloaded file."
 unzip $DOWNLOAD_DIR/$MLPLAN_ARC -d $TARGET_DIR
 
diff --git a/frameworks/RandomForest/__init__.py b/frameworks/RandomForest/__init__.py
index a25cbeee7..5d1f0aa49 100644
--- a/frameworks/RandomForest/__init__.py
+++ b/frameworks/RandomForest/__init__.py
@@ -23,8 +23,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/RandomForest/exec.py b/frameworks/RandomForest/exec.py
index 77bdc99ef..dd23a763d 100644
--- a/frameworks/RandomForest/exec.py
+++ b/frameworks/RandomForest/exec.py
@@ -3,6 +3,8 @@
 import tempfile as tmp
 from typing import List
 
+import pandas as pd
+
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
 os.environ['OMP_NUM_THREADS'] = '1'
 os.environ['OPENBLAS_NUM_THREADS'] = '1'
@@ -12,7 +14,7 @@
 import sklearn
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
 
-from frameworks.shared.callee import call_run, result
+from frameworks.shared.callee import call_run, result, measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(os.path.basename(__file__))
@@ -86,6 +88,19 @@ def run(dataset, config):
         predictions = rf.predict(X_test)
     probabilities = rf.predict_proba(X_test) if is_classification else None
 
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return rf.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+
     return result(output_file=config.output_predictions_file,
                   predictions=predictions,
                   truth=y_test,
@@ -93,7 +108,8 @@ def run(dataset, config):
                   target_is_encoded=encode,
                   models_count=len(rf),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,)
 
 
 if __name__ == '__main__':
diff --git a/frameworks/TPOT/__init__.py b/frameworks/TPOT/__init__.py
index 9828c6473..10b5ddb5f 100644
--- a/frameworks/TPOT/__init__.py
+++ b/frameworks/TPOT/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     def process_results(results):
         if isinstance(results.probabilities, str) and results.probabilities == "predictions":
diff --git a/frameworks/TPOT/exec.py b/frameworks/TPOT/exec.py
index ce70cb7f9..c6403d678 100644
--- a/frameworks/TPOT/exec.py
+++ b/frameworks/TPOT/exec.py
@@ -4,6 +4,9 @@
 import sys
 import tempfile as tmp
 
+import pandas as pd
+from numpy.random import default_rng
+
 if sys.platform == 'darwin':
     os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
@@ -13,7 +16,8 @@
 
 from tpot import TPOTClassifier, TPOTRegressor, __version__
 
-from frameworks.shared.callee import call_run, output_subdir, result
+from frameworks.shared.callee import call_run, output_subdir, result, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, is_sparse
 
 
@@ -62,15 +66,37 @@ def run(dataset, config):
     with Timer() as training:
         tpot.fit(X_train, y_train)
 
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        if is_classification:
+            try:
+                return tpot.predict_proba(data)
+            except (RuntimeError, AttributeError):
+                return tpot.predict(data)
+        return tpot.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        log.info("TPOT inference time measurements exclude preprocessing time of AMLB.")
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer, [
+                (1, dataset.test.X[default_rng(seed=i).integers(len(dataset.test.X)), :].reshape(1, -1))
+                for i in range(100)
+            ],
+        )
+
     log.info('Predicting on the test set.')
-    X_test = dataset.test.X
     y_test = dataset.test.y
     with Timer() as predict:
+        X_test = dataset.test.X
         predictions = tpot.predict(X_test)
+
     try:
         probabilities = tpot.predict_proba(X_test) if is_classification else None
-    except RuntimeError:
-        # TPOT throws a RuntimeError if the optimized pipeline does not support `predict_proba`.
+    except (RuntimeError, AttributeError):
+        # TPOT throws a RuntimeError or AttributeError if the optimized pipeline
+        # does not support `predict_proba` (which one depends on the version).
         probabilities = "predictions"  # encoding is handled by caller in `__init__.py`
 
     save_artifacts(tpot, config)
@@ -82,7 +108,9 @@ def run(dataset, config):
                   target_is_encoded=is_classification,
                   models_count=len(tpot.evaluated_individuals_),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,
+                  )
 
 
 def save_artifacts(estimator, config):
diff --git a/frameworks/TunedRandomForest/__init__.py b/frameworks/TunedRandomForest/__init__.py
index 561678497..b97439508 100644
--- a/frameworks/TunedRandomForest/__init__.py
+++ b/frameworks/TunedRandomForest/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/TunedRandomForest/exec.py b/frameworks/TunedRandomForest/exec.py
index 7c7a7dc15..c724487d0 100644
--- a/frameworks/TunedRandomForest/exec.py
+++ b/frameworks/TunedRandomForest/exec.py
@@ -21,7 +21,7 @@
 import sklearn
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
 
-from frameworks.shared.callee import call_run, result
+from frameworks.shared.callee import call_run, result, measure_inference_times
 from frameworks.shared.utils import Timer
 from custom_validate import cross_validate
 
@@ -211,6 +211,19 @@ def run(dataset, config):
         predictions = rf.predict(X_test)
     probabilities = rf.predict_proba(X_test) if is_classification else None
 
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return rf.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+
     return result(
         output_file=config.output_predictions_file,
         predictions=predictions,
@@ -219,7 +232,8 @@ def run(dataset, config):
         target_is_encoded=is_classification,
         models_count=len(rf),
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/autosklearn/__init__.py b/frameworks/autosklearn/__init__.py
index 3e31b6d64..637a81491 100644
--- a/frameworks/autosklearn/__init__.py
+++ b/frameworks/autosklearn/__init__.py
@@ -10,19 +10,23 @@ def setup(*args, **kwargs):
 def run(dataset: Dataset, config: TaskConfig):
     from frameworks.shared.caller import run_in_venv
 
-    X_train, X_test = dataset.train.X_enc, dataset.test.X_enc
-    y_train, y_test = unsparsify(dataset.train.y_enc, dataset.test.y_enc)
     data = dict(
         train=dict(
-            X=X_train,
-            y=y_train
+            X=dataset.train.X,
+            y=dataset.train.y,
+            X_enc=dataset.train.X_enc,
+            y_enc=unsparsify(dataset.train.y_enc),
         ),
         test=dict(
-            X=X_test,
-            y=y_test
+            X=dataset.test.X,
+            y=dataset.test.y,
+            X_enc=dataset.test.X_enc,
+            y_enc=unsparsify(dataset.test.y_enc),
         ),
-        predictors_type=['Numerical' if p.is_numerical() else 'Categorical' for p in dataset.predictors]
+        predictors_type=['Numerical' if p.is_numerical() else 'Categorical' for p in dataset.predictors],
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/autosklearn/exec.py b/frameworks/autosklearn/exec.py
index 11690e09c..6c98fc199 100644
--- a/frameworks/autosklearn/exec.py
+++ b/frameworks/autosklearn/exec.py
@@ -4,6 +4,10 @@
 import shutil
 import tempfile as tmp
 import warnings
+from typing import Union
+
+import pandas as pd
+from numpy.random import default_rng
 
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
 os.environ['OMP_NUM_THREADS'] = '1'
@@ -15,7 +19,8 @@
 import autosklearn.metrics as metrics
 from packaging import version
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, system_memory_mb, walk_apply, zip_path
 
 log = logging.getLogger(__name__)
@@ -62,8 +67,9 @@ def run(dataset, config):
     )
     log.info("Environment: %s", os.environ)
 
-    X_train = dataset.train.X
-    y_train = dataset.train.y
+    use_pandas = askl_version >= version.parse("0.15")
+    X_train = dataset.train.X if use_pandas else dataset.train.X_enc
+    y_train = dataset.train.y if use_pandas else dataset.train.y_enc
     predictors_type = dataset.predictors_type
     log.debug("predictors_type=%s", predictors_type)
 
@@ -118,6 +124,10 @@ def run(dataset, config):
     else:
         fit_extra_params['metric'] = perf_metric
 
+    if not use_pandas:
+        fit_extra_params["feat_type"] = predictors_type
+
+
     constr_params["time_left_for_this_task"] = config.max_runtime_seconds
     constr_params["n_jobs"] = n_jobs
     constr_params["seed"] = config.seed
@@ -128,13 +138,30 @@ def run(dataset, config):
 
     auto_sklearn = estimator(**constr_params, **training_params)
     with Timer() as training:
-        auto_sklearn.fit(X_train, y_train, feat_type=predictors_type, **fit_extra_params)
+        auto_sklearn.fit(X_train, y_train, **fit_extra_params)
+
+    def infer(data: Union[str, pd.DataFrame]):
+        test_data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = auto_sklearn.predict_proba if is_classification else auto_sklearn.predict
+        return predict_fn(test_data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = dataset.test.X if use_pandas else dataset.test.X_enc
+        def sample_one_test_row(seed: int):
+            if use_pandas:
+                return test_data.sample(1, random_state=seed)
+            return test_data[default_rng(seed=seed).integers(len(test_data)), :]
+
+        inference_times["df"] = measure_inference_times(
+            infer, [(1, sample_one_test_row(seed=i)) for i in range(100)],
+        )
 
     # Convert output to strings for classification
     log.info("Predicting on the test set.")
-    X_test = dataset.test.X
-    y_test = dataset.test.y
     with Timer() as predict:
+        X_test = dataset.test.X if use_pandas else dataset.test.X_enc
         predictions = auto_sklearn.predict(X_test)
     probabilities = auto_sklearn.predict_proba(X_test) if is_classification else None
 
@@ -142,12 +169,14 @@ def run(dataset, config):
 
     return result(output_file=config.output_predictions_file,
                   predictions=predictions,
-                  truth=y_test,
+                  truth=dataset.test.y if use_pandas else dataset.test.y_enc,
                   probabilities=probabilities,
-                  target_is_encoded=is_classification,
+                  target_is_encoded=is_classification and not use_pandas,
                   models_count=len(auto_sklearn.get_models_with_weights()),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,
+                  )
 
 
 def save_artifacts(estimator, config):
diff --git a/frameworks/autosklearn/requirements.txt b/frameworks/autosklearn/requirements.txt
index 6d4ac2d7d..9e7bca5c3 100644
--- a/frameworks/autosklearn/requirements.txt
+++ b/frameworks/autosklearn/requirements.txt
@@ -1,3 +1,3 @@
-packaging
+packaging<22.0
 openml
-scipy>=0.14.1,<1.7.0
+pandas<2.0  # https://github.com/automl/auto-sklearn/issues/1672
diff --git a/frameworks/autosklearn/setup.sh b/frameworks/autosklearn/setup.sh
index f0099c3e3..63edcf04f 100755
--- a/frameworks/autosklearn/setup.sh
+++ b/frameworks/autosklearn/setup.sh
@@ -36,6 +36,7 @@ else
     git clone  --recurse-submodules --shallow-submodules ${DEPTH} ${REPO} ${TARGET_DIR}
     cd ${TARGET_DIR}
     git checkout "${COMMIT}"
+    git submodule update --init --recursive
     cd ${HERE}
     PIP install -U -e ${TARGET_DIR}
 fi
diff --git a/frameworks/autoxgboost/__init__.py b/frameworks/autoxgboost/__init__.py
deleted file mode 100644
index e77eab6f7..000000000
--- a/frameworks/autoxgboost/__init__.py
+++ /dev/null
@@ -1,10 +0,0 @@
-from amlb.utils import call_script_in_same_dir
-
-
-def setup(*args, **kwargs):
-    call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
-
-
-def run(*args, **kwargs):
-    from .exec import run
-    return run(*args, **kwargs)
diff --git a/frameworks/autoxgboost/exec.R b/frameworks/autoxgboost/exec.R
deleted file mode 100644
index 08c54f90f..000000000
--- a/frameworks/autoxgboost/exec.R
+++ /dev/null
@@ -1,47 +0,0 @@
-library(mlr)
-library(autoxgboost)
-library(farff)
-
-run <- function(train_file, test_file, target.index, type, output_predictions_file, cores, time.budget, meta_results_file) {
-  train <- farff::readARFF(train_file)
-  colnames(train) <- make.names(colnames(train))
-  target <- colnames(train)[target.index]
-
-  if (type == "classification") {
-    train <- makeClassifTask(data = train, target = target)
-    lrn <- makeLearner("classif.autoxgboost", time.budget = time.budget, nthread = cores, predict.type = "prob")
-  } else if (type == "regression") {
-    train <- makeRegrTask(data = train, target = target)
-    lrn <- makeLearner("regr.autoxgboost", time.budget = time.budget, nthread = cores)
-  } else {
-    stop("Task type not supported!")
-  }
-
-  mod <- NULL
-  preds <- NULL
-  training <- function() mod <<- train(lrn, train)
-  prediction <- function() preds <<- predict(mod, newdata = test)$data
-
-  train_duration <- system.time(training())[['elapsed']]
-
-  test <- farff::readARFF(test_file)
-  colnames(test) <- make.names(colnames(test))
-  predict_duration <- system.time(prediction())[['elapsed']]
-
-  preds <- preds[c(2:ncol(preds), 1)]
-  names(preds)[names(preds) == "response"] <- "predictions"
-
-  if (type == "classification") {
-    names(preds) <- sub("^prob.", "", names(preds))
-  }
-  # FIXME: label encoding for predictions and truth?
-
-  write.csv(preds, file = output_predictions_file, row.names = FALSE)
-
-  meta_results <- data.frame(key=c("training_duration", "predict_duration"),
-                             value=c(train_duration, predict_duration))
-  write.csv(meta_results, file = meta_results_file, row.names = FALSE)
-}
-
-# args = commandArgs(trailingOnly=TRUE)
-# run(args[1], args[2], args[3], as.integer(args[4]))
diff --git a/frameworks/autoxgboost/exec.py b/frameworks/autoxgboost/exec.py
deleted file mode 100644
index a4ac54799..000000000
--- a/frameworks/autoxgboost/exec.py
+++ /dev/null
@@ -1,49 +0,0 @@
-import logging
-import os
-
-from amlb.benchmark import TaskConfig
-from amlb.data import Dataset
-from amlb.datautils import read_csv
-from amlb.utils import dir_of, run_cmd
-
-log = logging.getLogger(__name__)
-
-
-def run(dataset: Dataset, config: TaskConfig):
-    #TODO: use rpy2 instead? not necessary here though as the call is very simple
-    log.info(f"\n**** Autoxgboost (R) [{config.framework_version}] ****\n")
-
-    is_classification = config.type == 'classification'
-
-    here = dir_of(__file__)
-
-    meta_results_file = os.path.join(config.output_dir, "meta_results.csv")
-    run_cmd(("Rscript --vanilla -e \""
-             ".libPaths('{package_directory}'); source('{script}'); "
-             "run('{train}', '{test}', target.index = {target_index}, '{type}', '{output}', {cores},"
-             " time.budget = {time_budget}, meta_results_file='{meta_results}')"
-             "\"").format(
-        package_directory=os.path.join(here, "lib"),
-        script=os.path.join(here, 'exec.R'),
-        train=dataset.train.path,
-        test=dataset.test.path,
-        target_index=dataset.target.index+1,
-        type=config.type,
-        output=config.output_predictions_file,
-        cores=config.cores,
-        time_budget=config.max_runtime_seconds,
-        meta_results=meta_results_file
-    ), _live_output_=True)
-
-    log.info("Predictions saved to %s", config.output_predictions_file)
-
-    meta_results = read_csv(meta_results_file)
-    return dict(
-        training_duration=meta_result(meta_results, 'training_duration'),
-        predict_duration=meta_result(meta_results, 'predict_duration')
-    )
-
-
-def meta_result(df, key):
-    return df.loc[df['key'] == key, 'value'].squeeze()
-
diff --git a/frameworks/autoxgboost/requirements.txt b/frameworks/autoxgboost/requirements.txt
deleted file mode 100644
index 9ac4fbb1b..000000000
--- a/frameworks/autoxgboost/requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-rpy2==2.3.0
\ No newline at end of file
diff --git a/frameworks/autoxgboost/setup.sh b/frameworks/autoxgboost/setup.sh
deleted file mode 100755
index f73244502..000000000
--- a/frameworks/autoxgboost/setup.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/usr/bin/env bash
-HERE=$(dirname "$0")
-VERSION=${1:-"stable"}
-REPO=${2:-"ja-thomas/autoxgboost"}
-
-# Version can be specified as 'stable', 'latest', a branch or a commit hash (indicated by starting with '#')
-if [[ "$VERSION" == "latest" || "$VERSION" == "stable" ]]; then
-  VERSION="master"
-fi
-
-if [[ "$VERSION" =~ ^# ]]; then
-  VERSION="${VERSION:1}"
-else
-  # if VERSION is not a hash, it should be a branch (or a format which is not (officially) supported)
-  COMMIT=$(git ls-remote "https://github.com/${REPO}" | grep "refs/heads/${VERSION}" | cut -f 1)
-  if [[ -z $COMMIT ]]; then
-    echo "Could not resolve version ${VERSION}. It is not a branch on https://github.com/${REPO}."
-    echo "Continuing setup, install_github will try to resolve 'ref=${VERSION}'."
-  else
-    VERSION=$COMMIT
-  fi
-fi
-
-. ${HERE}/../shared/setup.sh "${HERE}"
-if [[ -x "$(command -v apt-get)" ]]; then
-  SUDO apt-get update
-  SUDO apt-get install -y software-properties-common dirmngr
-  SUDO apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
-  SUDO add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
-  SUDO apt-get update
-  SUDO apt-get install -y r-base r-base-dev
-  SUDO apt-get install -y libgdal-dev libproj-dev
-  SUDO apt-get install -y libssl-dev libcurl4-openssl-dev
-  SUDO apt-get install -y libcairo2-dev libudunits2-dev
-fi
-
-# We install the packages to a subdirectory in the framework folder, similar to venvs for Python, because:
-# - we want to be able to install different package versions for different frameworks in one local installation
-# - the default package directory is not always writeable (e.g. on Github CI)
-LIB="${HERE}/lib/"
-mkdir ${LIB}
-
-Rscript -e 'options(install.packages.check.source="no"); install.packages(c("remotes", "mlr", "mlrMBO", "mlrCPO", "farff", "GenSA", "rgenoud", "xgboost"), repos="https://cloud.r-project.org/", lib="'"${LIB}"'")'
-Rscript -e '.libPaths("'"${LIB}"'"); remotes::install_github("'"${REPO}"'", ref="'"${VERSION}"'", lib="'"${LIB}/"'")'
-
-OFFICIAL_VERSION=$(Rscript -e '.libPaths("'"${LIB}"'"); packageVersion("autoxgboost")' | awk '{print $2}' | sed "s/[‘’]//g")
-echo "${OFFICIAL_VERSION}#${VERSION:0:7}" >> "${HERE}/.setup/installed"
diff --git a/frameworks/constantpredictor/exec.py b/frameworks/constantpredictor/exec.py
index 6a7aae69c..049e19b76 100644
--- a/frameworks/constantpredictor/exec.py
+++ b/frameworks/constantpredictor/exec.py
@@ -1,11 +1,13 @@
 import logging
 
+import pandas as pd
 from sklearn.dummy import DummyClassifier, DummyRegressor
 
 from amlb.benchmark import TaskConfig
 from amlb.data import Dataset
 from amlb.results import save_predictions
 from amlb.utils import Timer, unsparsify
+from frameworks.shared.callee import measure_inference_times
 
 log = logging.getLogger(__name__)
 
@@ -29,6 +31,19 @@ def run(dataset: Dataset, config: TaskConfig):
         predictions = predictor.predict(X_test)
     probabilities = predictor.predict_proba(X_test) if is_classification else None
 
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return predictor.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files(fmt="parquet"))
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+
     save_predictions(dataset=dataset,
                      output_file=config.output_predictions_file,
                      probabilities=probabilities,
@@ -39,5 +54,6 @@ def run(dataset: Dataset, config: TaskConfig):
     return dict(
         models_count=1,
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
diff --git a/frameworks/flaml/__init__.py b/frameworks/flaml/__init__.py
index c911edf3b..dcec90325 100644
--- a/frameworks/flaml/__init__.py
+++ b/frameworks/flaml/__init__.py
@@ -17,8 +17,10 @@ def run(dataset, config):
             X=dataset.test.X,
             y=dataset.test.y
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/flaml/exec.py b/frameworks/flaml/exec.py
index 7c450ef47..510f02b9c 100644
--- a/frameworks/flaml/exec.py
+++ b/frameworks/flaml/exec.py
@@ -1,9 +1,12 @@
 import logging
 import os
+from typing import Union
 
+import pandas as pd
 from flaml import AutoML, __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(__name__)
@@ -13,7 +16,6 @@ def run(dataset, config):
     log.info(f"\n**** FLAML [v{__version__}] ****\n")
 
     X_train, y_train = dataset.train.X, dataset.train.y.squeeze()
-    X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
 
     is_classification = config.type == 'classification'
     time_budget = config.max_runtime_seconds
@@ -49,11 +51,27 @@ def run(dataset, config):
                 n_jobs=n_jobs,
                 log_file_name= flaml_log_file_name,
                 time_budget=time_budget, **training_params)
-    
+
+    def infer(data: Union[str, pd.DataFrame]):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = aml.predict_proba if is_classification else aml.predict
+        return predict_fn(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
         predictions = aml.predict(X_test)
     probabilities = aml.predict_proba(X_test) if is_classification else None
-    labels = aml.classes_ if is_classification else None
+    labels = None
+    if is_classification:
+        labels = aml.classes_ if isinstance(aml.classes_, list) else aml.classes_.tolist()
     return result(  
                     output_file=config.output_predictions_file,
                     probabilities=probabilities,
@@ -63,6 +81,7 @@ def run(dataset, config):
                     training_duration=training.duration,
                     predict_duration=predict.duration,
                     probabilities_labels=labels,
+                    inference_times=inference_times,
                 )
 
 
diff --git a/frameworks/flaml/setup.sh b/frameworks/flaml/setup.sh
index 776620157..7fb003789 100755
--- a/frameworks/flaml/setup.sh
+++ b/frameworks/flaml/setup.sh
@@ -3,22 +3,34 @@ HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/microsoft/FLAML.git"}
 PKG=${3:-"flaml"}
+
+
+. ${HERE}/../shared/setup.sh ${HERE} true
+
 if [[ "$VERSION" == "latest" ]]; then
     VERSION="main"
+    OPTIONALS=""
+fi
+
+if [[ "$VERSION" == "benchmark" ]]; then
+    VERSION="stable"
+    OPTIONALS="[catboost]"
+else
+    PIP uninstall -y catboost
 fi
 
-. ${HERE}/../shared/setup.sh ${HERE} true
 
 if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
+    PIP install --no-cache-dir -U "${PKG}${OPTIONALS}<2"
 elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+    PIP install --no-cache-dir -U ${PKG}${OPTIONALS}==${VERSION}
 else
 #    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg=${PKG}
     TARGET_DIR="${HERE}/lib/${PKG}"
     rm -Rf ${TARGET_DIR}
     git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
-    PIP install -U -e ${TARGET_DIR}
+    PIP install -U -e ${TARGET_DIR}${OPTIONALS}
 fi
 
 PY -c "from flaml import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+echo ${OPTIONALS} >> "${HERE}/.setup/installed"
diff --git a/frameworks/hyperoptsklearn/setup.sh b/frameworks/hyperoptsklearn/setup.sh
index c574f1848..b2cc19005 100755
--- a/frameworks/hyperoptsklearn/setup.sh
+++ b/frameworks/hyperoptsklearn/setup.sh
@@ -3,21 +3,27 @@ HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/hyperopt/hyperopt-sklearn.git"}
 PKG=${3:-"hyperopt-sklearn"}
-if [[ "$VERSION" == "latest" ]]; then
+
+if [[ "$VERSION" == "latest" || "$VERSION" == "stable" ]]; then
     VERSION="master"
 fi
 
 . ${HERE}/../shared/setup.sh ${HERE} true
 
-if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
-elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+if [[ "$VERSION" =~ ^[0-9] ]]; then
+    PIP install git+https://github.com/hyperopt/hyperopt-sklearn@${VERSION}
 else
 #    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg=${PKG}
     LIB=$(echo ${PKG} | sed "s/\[.*\]//")
     TARGET_DIR="${HERE}/lib/${LIB}"
     rm -Rf ${TARGET_DIR}
-    git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
+    if [[ "$VERSION" =~ ^# ]]; then
+        git clone --recurse-submodules --shallow-submodules ${REPO} ${TARGET_DIR}
+        cd ${TARGET_DIR}
+        git checkout "${VERSION:1}"
+        cd ${HERE}
+    else
+        git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
+    fi
     PIP install -U -e ${HERE}/lib/${PKG}
 fi
diff --git a/frameworks/lightautoml/__init__.py b/frameworks/lightautoml/__init__.py
index 97c09fa0e..4c9654850 100644
--- a/frameworks/lightautoml/__init__.py
+++ b/frameworks/lightautoml/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         target=dict(
             name=dataset.target.name,
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/lightautoml/exec.py b/frameworks/lightautoml/exec.py
index aee255902..01217abd1 100644
--- a/frameworks/lightautoml/exec.py
+++ b/frameworks/lightautoml/exec.py
@@ -2,16 +2,20 @@
 import os
 import pickle
 import warnings
+from typing import Union
 
 import matplotlib
 import numpy as np
+import pandas as pd
+
 matplotlib.use("agg")  # no need for tk
 
 from lightautoml.tasks import Task
 from lightautoml.automl.presets.tabular_presets import TabularAutoML, TabularUtilizedAutoML
 from lightautoml import __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(__name__)
@@ -37,9 +41,21 @@ def run(dataset, config):
     with Timer() as training:
         automl.fit_predict(train_data=df_train, roles={'target': label})
 
-    X_test, y_test = dataset.test.X, dataset.test.y
+    def infer(data: Union[str, pd.DataFrame]):
+        batch = pd.read_parquet(data) if isinstance(data, str) else data
+        return automl.predict(batch)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+
     log.info("Predicting on the test set...")
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y
         preds = automl.predict(X_test).data
 
     probabilities_labels = None
@@ -75,6 +91,7 @@ def run(dataset, config):
         predictions=predictions,
         training_duration=training.duration,
         predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/mljarsupervised/__init__.py b/frameworks/mljarsupervised/__init__.py
index 9bee9f4a5..b15d780f8 100644
--- a/frameworks/mljarsupervised/__init__.py
+++ b/frameworks/mljarsupervised/__init__.py
@@ -19,8 +19,10 @@ def run(dataset: Dataset, config: TaskConfig):
             X=dataset.test.X,
             y=dataset.test.y
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/mljarsupervised/exec.py b/frameworks/mljarsupervised/exec.py
index 653d9cfd6..66d2c6d64 100644
--- a/frameworks/mljarsupervised/exec.py
+++ b/frameworks/mljarsupervised/exec.py
@@ -1,15 +1,19 @@
 import os
 import shutil
 import logging
+from typing import Union
 
 import numpy as np
 import matplotlib
+import pandas as pd
+
 matplotlib.use("agg")  # no need for tk
 
 import supervised
 from supervised.automl import AutoML
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(os.path.basename(__file__))
@@ -42,7 +46,6 @@ def run(dataset, config):
     }
 
     X_train, y_train = dataset.train.X, dataset.train.y.squeeze()
-    X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
 
     automl = AutoML(
         results_path=results_path,
@@ -56,7 +59,21 @@ def run(dataset, config):
     with Timer() as training:
         automl.fit(X_train, y_train)
 
+
+    def infer(data: Union[str, pd.DataFrame]):
+        batch = pd.read_parquet(data) if isinstance(data, str) else data
+        return automl.predict_all(batch)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
         preds = automl.predict_all(X_test)
 
     predictions, probabilities, probabilities_labels = None, None, None
@@ -88,7 +105,8 @@ def run(dataset, config):
         probabilities_labels=probabilities_labels,
         models_count=len(automl._models),
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/mlr3automl/setup.sh b/frameworks/mlr3automl/setup.sh
index cd5225c04..b939770e4 100755
--- a/frameworks/mlr3automl/setup.sh
+++ b/frameworks/mlr3automl/setup.sh
@@ -24,7 +24,7 @@ fi
 
 . $HERE/../shared/setup.sh "$HERE"
 if [[ -x "$(command -v apt-get)" ]]; then
-SUDO apt-get update
+  SUDO apt-get update
   SUDO apt-get install -y software-properties-common dirmngr
   SUDO apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
   SUDO add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
diff --git a/frameworks/oboe/exec.py b/frameworks/oboe/exec.py
index 37feebe06..bed388dc7 100644
--- a/frameworks/oboe/exec.py
+++ b/frameworks/oboe/exec.py
@@ -2,8 +2,11 @@
 import os
 import sys
 
+from sklearn.model_selection import StratifiedKFold
+import numpy as np
+
 sys.path.append("{}/lib/oboe/automl".format(os.path.realpath(os.path.dirname(__file__))))
-from auto_learner import AutoLearner
+from oboe import AutoLearner
 
 from frameworks.shared.callee import call_run, result
 from frameworks.shared.utils import Timer
@@ -11,9 +14,46 @@
 log = logging.getLogger(__name__)
 
 
+def kfold_fit_validate(self, x_train, y_train, n_folds, random_state=None):
+    """Performs k-fold cross validation on a training dataset. Note that this is the function used to fill entries
+    of the error matrix.
+    Args:
+        x_train (np.ndarray): Features of the training dataset.
+        y_train (np.ndarray): Labels of the training dataset.
+        n_folds (int):        Number of folds to use for cross validation.
+    Returns:
+        float: Mean of k-fold cross validation error.
+        np.ndarray: Predictions on the training dataset from cross validation.
+    """
+    y_predicted = np.empty(y_train.shape)
+    cv_errors = np.empty(n_folds)
+    kf = StratifiedKFold(n_folds, shuffle=True, random_state=random_state)
+    for i, (train_idx, test_idx) in enumerate(kf.split(x_train, y_train)):
+        x_tr = x_train[train_idx, :]
+        y_tr = y_train[train_idx]
+        x_te = x_train[test_idx, :]
+        y_te = y_train[test_idx]
+        model = self.instantiate()
+        if len(np.unique(y_tr)) > 1:
+            model.fit(x_tr, y_tr)
+            y_predicted[test_idx] = np.expand_dims(model.predict(x_te), axis=1)
+        else:
+            y_predicted[test_idx] = y_tr[0]
+        cv_errors[i] = self.error(y_te, y_predicted[test_idx])
+    self.cv_error = cv_errors.mean()
+    self.cv_predictions = y_predicted
+    self.sampled = True
+    if self.verbose:
+        print("{} {} complete.".format(self.algorithm, self.hyperparameters))
+    return cv_errors, y_predicted
+
+
 def run(dataset, config):
-    log.info(f"\n**** Oboe [{config.framework_version}] ****\n")
+    log.info(f"\n**** Applying monkey patch ****\n")
+    from oboe.model import Model
+    Model.kfold_fit_validate = kfold_fit_validate
 
+    log.info(f"\n**** Oboe [{config.framework_version}] ****\n")
     is_classification = config.type == 'classification'
     if not is_classification:
         # regression currently fails (as of 26.02.2019: still under development state by oboe team)
@@ -32,6 +72,7 @@ def run(dataset, config):
                       n_cores=n_cores,
                       runtime_limit=config.max_runtime_seconds,
                       **training_params)
+    aml.error_matrix = aml.error_matrix.to_numpy()
 
     aml_models = lambda: [aml.ensemble, *aml.ensemble.base_learners] if len(aml.ensemble.base_learners) > 0 else []
 
diff --git a/frameworks/oboe/requirements.txt b/frameworks/oboe/requirements.txt
deleted file mode 100644
index a489529ca..000000000
--- a/frameworks/oboe/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-cvxpy>=1.0,<2.0
-numpy==1.16.4
-scipy==1.4.1
-pandas==0.24.2
-scikit-learn==0.22.1
-multiprocess>=0.70.5
-openml==0.10.2
-mkl>=1.0.0
-tensorly
diff --git a/frameworks/oboe/setup.sh b/frameworks/oboe/setup.sh
index 212bf9ffa..016d07241 100755
--- a/frameworks/oboe/setup.sh
+++ b/frameworks/oboe/setup.sh
@@ -21,5 +21,5 @@ else
     PIP install -U -e ${TARGET_DIR}
 fi
 
-cat ${HERE}/requirements.txt | sed '/^$/d' | while read -r i; do PIP install --no-cache-dir -U "$i"; done
+#cat ${HERE}/requirements.txt | sed '/^$/d' | while read -r i; do PIP install --no-cache-dir -U "$i"; done
 #PIP install --no-cache-dir -U -e git+https://github.com/udellgroup/oboe.git@${VERSION}#egg=oboe
diff --git a/frameworks/ranger/exec.R b/frameworks/ranger/exec.R
index a998badeb..5259cfed0 100644
--- a/frameworks/ranger/exec.R
+++ b/frameworks/ranger/exec.R
@@ -11,16 +11,17 @@ run <- function(train_file, test_file, output_predictions_file, cores=1, meta_re
   colnames(train) <- make.names(colnames(train))
   colnames(test) <- make.names(colnames(test))
   target <- colnames(train)[ncol(train)]
-  is_classifaction <- task_type == "classification"
-  task_class <- if (is_classifaction) TaskClassif else TaskRegr
+  is_classification <- task_type == "classification"
+  task_class <- if (is_classification) TaskClassif else TaskRegr
   train_task <- task_class$new(id="ranger.benchmark.train", backend = train, target = target)
   test_task <- task_class$new(id="ranger.benchmark.test", backend = test, target = target)
 
   learner <- po("removeconstants") %>>%
          po("imputeoor") %>>%
-         po("learner", learner = lrn(if (is_classifaction) "classif.ranger" else "regr.ranger",
-                                     num.threads = cores,
-                                     predict_type = if (is_classifaction) "prob" else "response"))
+         po("learner",
+            learner = lrn(if (is_classification) "classif.ranger" else "regr.ranger",
+                          num.threads = cores,
+                          predict_type = if (is_classification) "prob" else "response"))
 
   mod <- NULL
   preds <- NULL
@@ -28,7 +29,7 @@ run <- function(train_file, test_file, output_predictions_file, cores=1, meta_re
   prediction <- function() preds <<- learner$predict(test_task)[[mod]]
   train_duration <- system.time(training())[['elapsed']]
   predict_duration <- system.time(prediction())[['elapsed']]
-  if (is_classifaction) {
+  if (is_classification) {
     labels <- colnames(preds$prob)
     as_label <- function(x) labels[[x]]
     predictions <- cbind(preds$prob,
diff --git a/frameworks/ranger/exec.py b/frameworks/ranger/exec.py
index 3fb6905de..1ad60705e 100644
--- a/frameworks/ranger/exec.py
+++ b/frameworks/ranger/exec.py
@@ -14,11 +14,12 @@ def run(dataset: Dataset, config: TaskConfig):
     log.info("\n**** Random Forest (R) ****\n")
 
     here = dir_of(__file__)
-    meta_results_file = os.path.join(config.output_dir, "meta_results.csv")
+    meta_results_file = os.path.join(os.path.dirname(config.output_metadata_file), "meta_results.csv")
     run_cmd(("Rscript --vanilla -e \""
-             "source('{script}'); "
+             ".libPaths('{package_directory}'); source('{script}'); "
              "run('{train}', '{test}', '{output}', cores={cores}, meta_results_file='{meta_results}', task_type='{task_type}')"
              "\"").format(
+        package_directory=os.path.join(here, "lib"),
         script=os.path.join(here, 'exec.R'),
         train=dataset.train.path,
         test=dataset.test.path,
diff --git a/frameworks/ranger/setup.sh b/frameworks/ranger/setup.sh
index f279b3ac5..d36837dc1 100755
--- a/frameworks/ranger/setup.sh
+++ b/frameworks/ranger/setup.sh
@@ -15,7 +15,8 @@ if [[ -x "$(command -v apt-get)" ]]; then
     SUDO apt-get install -y r-base r-base-dev
 fi
 #PIP install --no-cache-dir -r $HERE/requirements.txt
+LIB="${HERE}/lib/"
+mkdir "${LIB}"
 
-Rscript -e 'options(install.packages.check.source="no"); install.packages(c("ranger", "mlr3", "mlr3learners", "mlr3pipelines", "farff"), repos="https://cloud.r-project.org/")'
-
+Rscript -e 'options(install.packages.check.source="no"); install.packages(c("ranger", "mlr3", "mlr3learners", "mlr3pipelines", "farff"), repos="https://cloud.r-project.org/", lib="'"${LIB}"'")'
 Rscript -e 'packageVersion("ranger")' | awk '{print $2}' | sed "s/[‘’]//g" >> "${HERE}/.setup/installed"
diff --git a/frameworks/shared/callee.py b/frameworks/shared/callee.py
index 3bf70dd3c..84fe1c6dc 100644
--- a/frameworks/shared/callee.py
+++ b/frameworks/shared/callee.py
@@ -1,11 +1,15 @@
 import logging
 import os
+import pathlib
 import re
 import signal
 import sys
+from collections import defaultdict
+from typing import Callable, Any, Tuple, Union, TypeVar, TYPE_CHECKING
 
 from .utils import InterruptTimeout, Namespace as ns, json_dump, json_loads, kill_proc_tree, touch
-from .utils import deserialize_data, serialize_data
+from .utils import deserialize_data, serialize_data, Timer
+
 
 log = logging.getLogger(__name__)
 
@@ -17,6 +21,7 @@ class FrameworkError(Exception):
 def result(output_file=None,
            predictions=None, truth=None,
            probabilities=None, probabilities_labels=None,
+           optional_columns=None,
            target_is_encoded=False,
            error_message=None,
            models_count=None,
@@ -69,7 +74,7 @@ def load_data(name, path, **_):
                               wait_retry_secs=10):
             result = run_fn(ds, config)
             res = dict(result)
-            for name in ['predictions', 'truth', 'probabilities']:
+            for name in ['predictions', 'truth', 'probabilities', 'optional_columns']:
                 arr = result[name]
                 if arr is not None:
                     path = os.path.join(config.result_dir, '.'.join([name, 'data']))
@@ -84,4 +89,25 @@ def load_data(name, path, **_):
         # ensure there's no subprocess left
         kill_proc_tree(include_parent=False, timeout=5)
 
+    inference_measurements = res.get("others", {}).get("inference_times")
+    if inference_measurements:
+        inference_file = pathlib.Path(config.result_file).parent / "inference_times.json"
+        json_dump(inference_measurements, inference_file, style="compact")
+        res["others"]["inference_times"] = str(inference_file)
     json_dump(res, config.result_file, style='compact')
+
+try:
+    import pandas as pd
+    DATA_TYPES = Union[str, pd.DataFrame]
+except ImportError:
+    DATA_TYPES = str
+
+DATA_INPUT = TypeVar("DATA_INPUT", bound=DATA_TYPES)
+
+def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:
+    inference_times = defaultdict(list)
+    for subsample_size, subsample_path in files:
+        with Timer() as predict:
+            predict_fn(subsample_path)
+        inference_times[subsample_size].append(predict.duration)
+    return inference_times
\ No newline at end of file
diff --git a/frameworks/shared/caller.py b/frameworks/shared/caller.py
index b4c9b0210..8422b98d4 100644
--- a/frameworks/shared/caller.py
+++ b/frameworks/shared/caller.py
@@ -1,9 +1,10 @@
 import gc
 import logging
 import os
+import pathlib
 import re
 from tempfile import TemporaryDirectory, mktemp
-from typing import Optional, Union
+from typing import List, Optional, Union
 
 import numpy as np
 
@@ -11,6 +12,7 @@
 from amlb.data import Dataset
 from amlb.resources import config as rconfig
 from amlb.results import NoResultError, save_predictions
+from amlb.utils import json_dump, Namespace
 
 from .utils import Namespace as ns, Timer, dir_of, run_cmd, json_dumps, json_load, profile
 from .utils import is_serializable_data, deserialize_data, serialize_data
@@ -82,7 +84,8 @@ def run_in_venv(caller_file, script_file: str, *args,
                 input_data: Union[dict, ns], dataset: Dataset, config: TaskConfig,
                 options: Union[None, dict, ns] = None,
                 process_results=None,
-                python_exec=None):
+                python_exec=None,
+                retained_env_vars: Optional[List[str]] = ['TMP', 'TEMP', 'TMPDIR']):
     here = dir_of(caller_file)
     if python_exec is None:  # use local virtual env by default
         python_exec = venv_python_exec(here)
@@ -93,6 +96,13 @@ def run_in_venv(caller_file, script_file: str, *args,
     ser_config = options['serialization']
     env = options['env'] or ns()
 
+    # Add any env variables specified if they are defined in the environment
+    if retained_env_vars:
+        for env_var in retained_env_vars:
+            env_val = os.environ.get(env_var)
+            if env_val is not None:
+                env[env_var] = env_val
+
     with TemporaryDirectory(prefix='amlb_', suffix='_xproc') as tmpdir:
 
         ds = _make_input_dataset(input_data, dataset, tmpdir, serialization=ser_config)
@@ -115,7 +125,7 @@ def run_in_venv(caller_file, script_file: str, *args,
             PYTHONPATH=os.pathsep.join([
                 rconfig().root_dir,
             ]),
-            AMLB_PATH=os.path.join(rconfig().root_dir, "amlb"),
+            AMLB_PATH=os.path.join(rconfig().root_dir),
             AMLB_LOG_TRACE=str(logging.TRACE if hasattr(logging, 'TRACE') else ''),
             **{k: str(v) for k, v in env}
         )
@@ -141,9 +151,16 @@ def run_in_venv(caller_file, script_file: str, *args,
         if res.error_message is not None:
             raise NoResultError(res.error_message)
 
-        for name in ['predictions', 'truth', 'probabilities']:
+        for name in ['predictions', 'truth', 'probabilities', 'optional_columns']:
             res[name] = deserialize_data(res[name], config=ser_config) if res[name] is not None else None
 
+        inference_filepath = Namespace.dict(res.others).get("inference_times")
+        if inference_filepath:
+            inference_times = json_load(inference_filepath)
+            inference_filepath = pathlib.Path(res.output_file).parent / "inference.json"
+            json_dump(inference_times, inference_filepath)
+            res["others"]["inference_times"] = inference_times
+
         if callable(process_results):
             res = process_results(res)
 
@@ -156,6 +173,7 @@ def run_in_venv(caller_file, script_file: str, *args,
                                     else dataset.test.y),
                              probabilities=res.probabilities,
                              probabilities_labels=res.probabilities_labels,
+                             optional_columns=res.optional_columns,
                              target_is_encoded=res.target_is_encoded)
 
         return dict(
diff --git a/frameworks/shared/requirements.in b/frameworks/shared/requirements.in
index b09b57f51..bfe9e1399 100644
--- a/frameworks/shared/requirements.in
+++ b/frameworks/shared/requirements.in
@@ -1,5 +1,4 @@
 psutil>=5.4
-ruamel.yaml>=0.15
 
 pyarrow>=4.0
-#tables>=3.6    # if using hdf serializer (let the client choose)
+# tables>=3.6    # if using hdf serializer (let the client choose)
diff --git a/frameworks/shared/requirements.txt b/frameworks/shared/requirements.txt
index 53569f1cb..4f291b10b 100644
--- a/frameworks/shared/requirements.txt
+++ b/frameworks/shared/requirements.txt
@@ -1,16 +1,11 @@
 #
-# This file is autogenerated by pip-compile
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
 #
-#    pip-compile frameworks/shared/requirements.in
+#    pip-compile --output-file=frameworks/shared/requirements.txt frameworks/shared/requirements.in
 #
-numpy==1.21.0
+numpy==1.24.2
     # via pyarrow
 psutil==5.8.0
     # via -r frameworks/shared/requirements.in
-pyarrow==4.0.1
-    # via -r frameworks/shared/requirements.in
-ruamel.yaml.clib==0.2.2
-    # via ruamel.yaml
-ruamel.yaml==0.17.4
-    # via -r frameworks/shared/requirements.in
+pyarrow==11.0.0
diff --git a/frameworks/shared/utils.py b/frameworks/shared/utils.py
index 23f2b6555..abcff3717 100644
--- a/frameworks/shared/utils.py
+++ b/frameworks/shared/utils.py
@@ -3,7 +3,6 @@
 import logging
 import os
 import sys
-from types import ModuleType
 
 
 def setup_logger():
@@ -20,6 +19,8 @@ def setup_logger():
 
 setup_logger()
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def load_module(name, path):
     spec = importlib.util.spec_from_file_location(name, path)
@@ -29,14 +30,20 @@ def load_module(name, path):
     return module
 
 
-def load_utils():
-    amlb_path = os.environ.get("AMLB_PATH")
+def load_amlb_module(mod, amlb_path=None):
+    amlb_path = amlb_path or os.environ.get("AMLB_PATH")
     if amlb_path:
-        return load_module("amlb.utils", os.path.join(amlb_path, "utils", "__init__.py"))
-    return import_module("amlb.utils")
+        tokens = mod.split('.')
+        mod_path = os.path.join(amlb_path, *tokens)
+        if os.path.isdir(mod_path):
+            tokens.append("__init__.py")
+            mod_path = os.path.join(amlb_path, *tokens)
+        return load_module(mod, mod_path)
+    return import_module(mod)
 
 
-utils = load_utils()
+utils = load_amlb_module("amlb.utils")
 # unorthodox for it's only now that we can safely import those functions
 from amlb.utils import *
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/reports/datasets/1h/metadata.csv b/reports/old/datasets/1h/metadata.csv
similarity index 100%
rename from reports/datasets/1h/metadata.csv
rename to reports/old/datasets/1h/metadata.csv
diff --git a/reports/datasets/4h/metadata.csv b/reports/old/datasets/4h/metadata.csv
similarity index 100%
rename from reports/datasets/4h/metadata.csv
rename to reports/old/datasets/4h/metadata.csv
diff --git a/reports/duration_comparison.ipynb b/reports/old/duration_comparison.ipynb
similarity index 100%
rename from reports/duration_comparison.ipynb
rename to reports/old/duration_comparison.ipynb
diff --git a/reports/graphics/1h/binary_norm_score_heat.png b/reports/old/graphics/1h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_heat.png
rename to reports/old/graphics/1h/binary_norm_score_heat.png
diff --git a/reports/graphics/1h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/1h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/1h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/1h/binary_norm_score_stripplot.png b/reports/old/graphics/1h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_stripplot.png
rename to reports/old/graphics/1h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/1h/binary_results_stripplot.png b/reports/old/graphics/1h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/1h/binary_results_stripplot.png
rename to reports/old/graphics/1h/binary_results_stripplot.png
diff --git a/reports/graphics/1h/binary_score_heat.png b/reports/old/graphics/1h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/1h/binary_score_heat.png
rename to reports/old/graphics/1h/binary_score_heat.png
diff --git a/reports/graphics/1h/binary_score_parallel_ccord.png b/reports/old/graphics/1h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/binary_score_parallel_ccord.png
rename to reports/old/graphics/1h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/1h/multiclass_norm_score_heat.png b/reports/old/graphics/1h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_heat.png
rename to reports/old/graphics/1h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/1h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/1h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/1h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/1h/multiclass_norm_score_stripplot.png b/reports/old/graphics/1h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/1h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/1h/multiclass_results_stripplot.png b/reports/old/graphics/1h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/1h/multiclass_results_stripplot.png
rename to reports/old/graphics/1h/multiclass_results_stripplot.png
diff --git a/reports/graphics/1h/multiclass_score_heat.png b/reports/old/graphics/1h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/1h/multiclass_score_heat.png
rename to reports/old/graphics/1h/multiclass_score_heat.png
diff --git a/reports/graphics/1h/multiclass_score_parallel_ccord.png b/reports/old/graphics/1h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/1h/multiclass_score_parallel_ccord.png
diff --git a/reports/graphics/4h/binary_norm_score_heat.png b/reports/old/graphics/4h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_heat.png
rename to reports/old/graphics/4h/binary_norm_score_heat.png
diff --git a/reports/graphics/4h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/4h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/4h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/4h/binary_norm_score_stripplot.png b/reports/old/graphics/4h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_stripplot.png
rename to reports/old/graphics/4h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/4h/binary_results_stripplot.png b/reports/old/graphics/4h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/4h/binary_results_stripplot.png
rename to reports/old/graphics/4h/binary_results_stripplot.png
diff --git a/reports/graphics/4h/binary_score_heat.png b/reports/old/graphics/4h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/4h/binary_score_heat.png
rename to reports/old/graphics/4h/binary_score_heat.png
diff --git a/reports/graphics/4h/binary_score_parallel_ccord.png b/reports/old/graphics/4h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/binary_score_parallel_ccord.png
rename to reports/old/graphics/4h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/4h/multiclass_norm_score_heat.png b/reports/old/graphics/4h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_heat.png
rename to reports/old/graphics/4h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/4h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/4h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/4h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/4h/multiclass_norm_score_stripplot.png b/reports/old/graphics/4h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/4h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/4h/multiclass_results_stripplot.png b/reports/old/graphics/4h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/4h/multiclass_results_stripplot.png
rename to reports/old/graphics/4h/multiclass_results_stripplot.png
diff --git a/reports/graphics/4h/multiclass_score_heat.png b/reports/old/graphics/4h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/4h/multiclass_score_heat.png
rename to reports/old/graphics/4h/multiclass_score_heat.png
diff --git a/reports/graphics/4h/multiclass_score_parallel_ccord.png b/reports/old/graphics/4h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/4h/multiclass_score_parallel_ccord.png
diff --git a/reports/graphics/8h/binary_norm_score_heat.png b/reports/old/graphics/8h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_heat.png
rename to reports/old/graphics/8h/binary_norm_score_heat.png
diff --git a/reports/graphics/8h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/8h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/8h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/8h/binary_norm_score_stripplot.png b/reports/old/graphics/8h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_stripplot.png
rename to reports/old/graphics/8h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/8h/binary_results_stripplot.png b/reports/old/graphics/8h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/8h/binary_results_stripplot.png
rename to reports/old/graphics/8h/binary_results_stripplot.png
diff --git a/reports/graphics/8h/binary_score_heat.png b/reports/old/graphics/8h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/8h/binary_score_heat.png
rename to reports/old/graphics/8h/binary_score_heat.png
diff --git a/reports/graphics/8h/binary_score_parallel_ccord.png b/reports/old/graphics/8h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/binary_score_parallel_ccord.png
rename to reports/old/graphics/8h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/8h/multiclass_norm_score_heat.png b/reports/old/graphics/8h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_heat.png
rename to reports/old/graphics/8h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/8h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/8h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/8h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/8h/multiclass_norm_score_stripplot.png b/reports/old/graphics/8h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/8h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/8h/multiclass_results_stripplot.png b/reports/old/graphics/8h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/8h/multiclass_results_stripplot.png
rename to reports/old/graphics/8h/multiclass_results_stripplot.png
diff --git a/reports/graphics/8h/multiclass_score_heat.png b/reports/old/graphics/8h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/8h/multiclass_score_heat.png
rename to reports/old/graphics/8h/multiclass_score_heat.png
diff --git a/reports/graphics/8h/multiclass_score_parallel_ccord.png b/reports/old/graphics/8h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/8h/multiclass_score_parallel_ccord.png
diff --git a/reports/input/results_large-8c4h.csv b/reports/old/input/results_large-8c4h.csv
similarity index 100%
rename from reports/input/results_large-8c4h.csv
rename to reports/old/input/results_large-8c4h.csv
diff --git a/reports/input/results_large-8c4h_ref.csv b/reports/old/input/results_large-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_large-8c4h_ref.csv
rename to reports/old/input/results_large-8c4h_ref.csv
diff --git a/reports/input/results_large-8c8h.csv b/reports/old/input/results_large-8c8h.csv
similarity index 100%
rename from reports/input/results_large-8c8h.csv
rename to reports/old/input/results_large-8c8h.csv
diff --git a/reports/input/results_large-8c8h_ref.csv b/reports/old/input/results_large-8c8h_ref.csv
similarity index 100%
rename from reports/input/results_large-8c8h_ref.csv
rename to reports/old/input/results_large-8c8h_ref.csv
diff --git a/reports/input/results_medium-4c1h.csv b/reports/old/input/results_medium-4c1h.csv
similarity index 100%
rename from reports/input/results_medium-4c1h.csv
rename to reports/old/input/results_medium-4c1h.csv
diff --git a/reports/input/results_medium-4c1h_ref.csv b/reports/old/input/results_medium-4c1h_ref.csv
similarity index 100%
rename from reports/input/results_medium-4c1h_ref.csv
rename to reports/old/input/results_medium-4c1h_ref.csv
diff --git a/reports/input/results_medium-4c4h.csv b/reports/old/input/results_medium-4c4h.csv
similarity index 100%
rename from reports/input/results_medium-4c4h.csv
rename to reports/old/input/results_medium-4c4h.csv
diff --git a/reports/input/results_medium-4c4h_ref.csv b/reports/old/input/results_medium-4c4h_ref.csv
similarity index 100%
rename from reports/input/results_medium-4c4h_ref.csv
rename to reports/old/input/results_medium-4c4h_ref.csv
diff --git a/reports/input/results_medium-8c1h.csv b/reports/old/input/results_medium-8c1h.csv
similarity index 100%
rename from reports/input/results_medium-8c1h.csv
rename to reports/old/input/results_medium-8c1h.csv
diff --git a/reports/input/results_medium-8c1h_ref.csv b/reports/old/input/results_medium-8c1h_ref.csv
similarity index 100%
rename from reports/input/results_medium-8c1h_ref.csv
rename to reports/old/input/results_medium-8c1h_ref.csv
diff --git a/reports/input/results_medium-8c4h.csv b/reports/old/input/results_medium-8c4h.csv
similarity index 100%
rename from reports/input/results_medium-8c4h.csv
rename to reports/old/input/results_medium-8c4h.csv
diff --git a/reports/input/results_medium-8c4h_ref.csv b/reports/old/input/results_medium-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_medium-8c4h_ref.csv
rename to reports/old/input/results_medium-8c4h_ref.csv
diff --git a/reports/input/results_old.csv b/reports/old/input/results_old.csv
similarity index 100%
rename from reports/input/results_old.csv
rename to reports/old/input/results_old.csv
diff --git a/reports/input/results_small-2c1h.csv b/reports/old/input/results_small-2c1h.csv
similarity index 100%
rename from reports/input/results_small-2c1h.csv
rename to reports/old/input/results_small-2c1h.csv
diff --git a/reports/input/results_small-2c1h_ref.csv b/reports/old/input/results_small-2c1h_ref.csv
similarity index 100%
rename from reports/input/results_small-2c1h_ref.csv
rename to reports/old/input/results_small-2c1h_ref.csv
diff --git a/reports/input/results_small-8c1h.csv b/reports/old/input/results_small-8c1h.csv
similarity index 100%
rename from reports/input/results_small-8c1h.csv
rename to reports/old/input/results_small-8c1h.csv
diff --git a/reports/input/results_small-8c1h_ref.csv b/reports/old/input/results_small-8c1h_ref.csv
similarity index 100%
rename from reports/input/results_small-8c1h_ref.csv
rename to reports/old/input/results_small-8c1h_ref.csv
diff --git a/reports/input/results_small-8c4h.csv b/reports/old/input/results_small-8c4h.csv
similarity index 100%
rename from reports/input/results_small-8c4h.csv
rename to reports/old/input/results_small-8c4h.csv
diff --git a/reports/input/results_small-8c4h_ref.csv b/reports/old/input/results_small-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_small-8c4h_ref.csv
rename to reports/old/input/results_small-8c4h_ref.csv
diff --git a/reports/input/results_valid.csv b/reports/old/input/results_valid.csv
similarity index 100%
rename from reports/input/results_valid.csv
rename to reports/old/input/results_valid.csv
diff --git a/reports/input/results_valid_ref.csv b/reports/old/input/results_valid_ref.csv
similarity index 100%
rename from reports/input/results_valid_ref.csv
rename to reports/old/input/results_valid_ref.csv
diff --git a/reports/old/readme.md b/reports/old/readme.md
new file mode 100644
index 000000000..5d6ae51d1
--- /dev/null
+++ b/reports/old/readme.md
@@ -0,0 +1,3 @@
+# Reports for 2019 AutoML Benchmark
+
+These notebooks are built based on a previous iteration of the AutoML benchmark output, so will not work out-of-the-box with the new result files.
diff --git a/reports/reports.ipynb b/reports/old/reports.ipynb
similarity index 100%
rename from reports/reports.ipynb
rename to reports/old/reports.ipynb
diff --git a/reports/tables/1h/all_results.csv b/reports/old/tables/1h/all_results.csv
similarity index 100%
rename from reports/tables/1h/all_results.csv
rename to reports/old/tables/1h/all_results.csv
diff --git a/reports/tables/1h/folds_leaderboard.csv b/reports/old/tables/1h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/1h/folds_leaderboard.csv
rename to reports/old/tables/1h/folds_leaderboard.csv
diff --git a/reports/tables/1h/models_summary.csv b/reports/old/tables/1h/models_summary.csv
similarity index 100%
rename from reports/tables/1h/models_summary.csv
rename to reports/old/tables/1h/models_summary.csv
diff --git a/reports/tables/1h/norm_score_summary.csv b/reports/old/tables/1h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/1h/norm_score_summary.csv
rename to reports/old/tables/1h/norm_score_summary.csv
diff --git a/reports/tables/1h/result_summary.csv b/reports/old/tables/1h/result_summary.csv
similarity index 100%
rename from reports/tables/1h/result_summary.csv
rename to reports/old/tables/1h/result_summary.csv
diff --git a/reports/tables/1h/score_summary.csv b/reports/old/tables/1h/score_summary.csv
similarity index 100%
rename from reports/tables/1h/score_summary.csv
rename to reports/old/tables/1h/score_summary.csv
diff --git a/reports/tables/1h/tasks_leaderboard.csv b/reports/old/tables/1h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/1h/tasks_leaderboard.csv
rename to reports/old/tables/1h/tasks_leaderboard.csv
diff --git a/reports/tables/4h/all_results.csv b/reports/old/tables/4h/all_results.csv
similarity index 100%
rename from reports/tables/4h/all_results.csv
rename to reports/old/tables/4h/all_results.csv
diff --git a/reports/tables/4h/folds_leaderboard.csv b/reports/old/tables/4h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/4h/folds_leaderboard.csv
rename to reports/old/tables/4h/folds_leaderboard.csv
diff --git a/reports/tables/4h/models_summary.csv b/reports/old/tables/4h/models_summary.csv
similarity index 100%
rename from reports/tables/4h/models_summary.csv
rename to reports/old/tables/4h/models_summary.csv
diff --git a/reports/tables/4h/norm_score_summary.csv b/reports/old/tables/4h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/4h/norm_score_summary.csv
rename to reports/old/tables/4h/norm_score_summary.csv
diff --git a/reports/tables/4h/result_summary.csv b/reports/old/tables/4h/result_summary.csv
similarity index 100%
rename from reports/tables/4h/result_summary.csv
rename to reports/old/tables/4h/result_summary.csv
diff --git a/reports/tables/4h/score_summary.csv b/reports/old/tables/4h/score_summary.csv
similarity index 100%
rename from reports/tables/4h/score_summary.csv
rename to reports/old/tables/4h/score_summary.csv
diff --git a/reports/tables/4h/tasks_leaderboard.csv b/reports/old/tables/4h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/4h/tasks_leaderboard.csv
rename to reports/old/tables/4h/tasks_leaderboard.csv
diff --git a/reports/tables/8h/all_results.csv b/reports/old/tables/8h/all_results.csv
similarity index 100%
rename from reports/tables/8h/all_results.csv
rename to reports/old/tables/8h/all_results.csv
diff --git a/reports/tables/8h/folds_leaderboard.csv b/reports/old/tables/8h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/8h/folds_leaderboard.csv
rename to reports/old/tables/8h/folds_leaderboard.csv
diff --git a/reports/tables/8h/models_summary.csv b/reports/old/tables/8h/models_summary.csv
similarity index 100%
rename from reports/tables/8h/models_summary.csv
rename to reports/old/tables/8h/models_summary.csv
diff --git a/reports/tables/8h/norm_score_summary.csv b/reports/old/tables/8h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/8h/norm_score_summary.csv
rename to reports/old/tables/8h/norm_score_summary.csv
diff --git a/reports/tables/8h/result_summary.csv b/reports/old/tables/8h/result_summary.csv
similarity index 100%
rename from reports/tables/8h/result_summary.csv
rename to reports/old/tables/8h/result_summary.csv
diff --git a/reports/tables/8h/score_summary.csv b/reports/old/tables/8h/score_summary.csv
similarity index 100%
rename from reports/tables/8h/score_summary.csv
rename to reports/old/tables/8h/score_summary.csv
diff --git a/reports/tables/8h/tasks_leaderboard.csv b/reports/old/tables/8h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/8h/tasks_leaderboard.csv
rename to reports/old/tables/8h/tasks_leaderboard.csv
diff --git a/reports/readme.md b/reports/readme.md
new file mode 100644
index 000000000..4dee98d67
--- /dev/null
+++ b/reports/readme.md
@@ -0,0 +1,4 @@
+# Notebooks for Visualizing AMLB Results
+
+This folder will contain the notebooks used to visualize result files produced by the AMLB tool, including those notebooks used for generating figures and tables in our paper.
+These notebooks will be added soon.
diff --git a/requirements.in b/requirements.in
index f4f3f772b..19adef7d2 100644
--- a/requirements.in
+++ b/requirements.in
@@ -1,10 +1,13 @@
 boto3>=1.9,<2.0
 liac-arff>=2.5,<3.0
-numpy>=1.20,<2.0
-pandas>=1.2.4,<2.0
 psutil>=5.4,<6.0
 ruamel.yaml>=0.15,<1.0
-openml==0.12.2
-scikit-learn>=0.24
-pyarrow>=4.0
-tables>=3.6
+filelock>=3.4
+
+numpy>=1.24,<2.0
+pandas>=1.5,<2.0
+openml==0.13.1
+scikit-learn>=1.0,<2.0
+
+pyarrow>=11.0
+# tables>=3.6
diff --git a/requirements.txt b/requirements.txt
index 2c2dd4a90..52d6e87a6 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,92 +1,88 @@
 #
-# This file is autogenerated by pip-compile
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
 #
-#    pip-compile requirements.in
+#    pip-compile --output-file=requirements.txt requirements.in
 #
-boto3==1.17.74
+boto3==1.26.98
     # via -r requirements.in
-botocore==1.20.74
+botocore==1.29.98
     # via
     #   boto3
     #   s3transfer
-certifi==2020.12.5
+certifi==2022.12.7
     # via
     #   minio
     #   requests
-chardet==4.0.0
+charset-normalizer==3.1.0
     # via requests
-idna==2.10
+filelock==3.12.0
+    # via -r requirements.in
+idna==3.4
     # via requests
-jmespath==0.10.0
+jmespath==1.0.1
     # via
     #   boto3
     #   botocore
-joblib==1.0.1
+joblib==1.2.0
     # via scikit-learn
 liac-arff==2.5.0
     # via
     #   -r requirements.in
     #   openml
-minio==7.0.3
+minio==7.1.13
     # via openml
-numexpr==2.7.3
-    # via tables
-numpy==1.20.3
+numpy==1.24.2
     # via
     #   -r requirements.in
-    #   numexpr
     #   openml
     #   pandas
     #   pyarrow
     #   scikit-learn
     #   scipy
-    #   tables
-openml==0.12.2
+openml==0.13.1
     # via -r requirements.in
-pandas==1.2.4
+pandas==1.5.3
     # via
     #   -r requirements.in
     #   openml
-psutil==5.8.0
+psutil==5.9.4
     # via -r requirements.in
-pyarrow==4.0.0
+pyarrow==11.0.0
     # via
     #   -r requirements.in
     #   openml
-python-dateutil==2.8.1
+python-dateutil==2.8.2
     # via
     #   botocore
     #   openml
     #   pandas
-pytz==2021.1
+pytz==2022.7.1
     # via pandas
-requests==2.25.1
+requests==2.28.2
     # via openml
-ruamel.yaml.clib==0.2.2
-    # via ruamel.yaml
-ruamel.yaml==0.17.4
+ruamel-yaml==0.17.21
     # via -r requirements.in
-s3transfer==0.4.2
+ruamel-yaml-clib==0.2.7
+    # via ruamel-yaml
+s3transfer==0.6.0
     # via boto3
-scikit-learn==0.24.2
+scikit-learn==1.2.2
     # via
     #   -r requirements.in
     #   openml
-scipy==1.6.3
+scipy==1.10.1
     # via
     #   openml
     #   scikit-learn
 six==1.16.0
     # via python-dateutil
-tables==3.6.1
-    # via -r requirements.in
-threadpoolctl==2.1.0
+threadpoolctl==3.1.0
     # via scikit-learn
-urllib3==1.26.4
+urllib3==1.26.15
     # via
     #   botocore
     #   minio
     #   requests
-xmltodict==0.12.0
+xmltodict==0.13.0
     # via openml
diff --git a/resources/benchmarks/timeseries.yaml b/resources/benchmarks/timeseries.yaml
new file mode 100644
index 000000000..26af06497
--- /dev/null
+++ b/resources/benchmarks/timeseries.yaml
@@ -0,0 +1,13 @@
+---
+
+- name: covid
+  dataset:
+    train: https://autogluon.s3-us-west-2.amazonaws.com/datasets/CovidTimeSeries/train.csv
+    test: https://autogluon.s3-us-west-2.amazonaws.com/datasets/CovidTimeSeries/test.csv
+    target: ConfirmedCases
+    type: timeseries
+    forecast_range_in_steps: 19
+    id_column: name
+    timestamp_column: Date
+
+  folds: 1
diff --git a/resources/config.yaml b/resources/config.yaml
index c03a742eb..c107f4412 100644
--- a/resources/config.yaml
+++ b/resources/config.yaml
@@ -33,7 +33,7 @@ frameworks:              # configuration namespace for the frameworks definition
     - '{root}/resources/frameworks.yaml'
   root_module: frameworks     # the default python module under which the frameworks modules are defined.
   allow_duplicates: false     # if true, the last definition is used.
-  tags: ['stable', 'latest', '2020Q2', '2021Q3']  # the list of supported tags when looking up frameworks:
+  tags: ['stable', 'latest', '2020Q2', '2021Q3', '2023Q2']  # the list of supported tags when looking up frameworks:
                               # for example frmwk:latest will look for framework frmwk in a frameworks_latest.yaml file if present.
 
 benchmarks:                     # configuration namespace for the benchmarks definitions.
@@ -54,6 +54,7 @@ benchmarks:                     # configuration namespace for the benchmarks def
     binary: ['auc', 'logloss', 'acc', 'balacc']     # available metrics: auc (AUC), acc (Accuracy), balacc (Balanced Accuracy), pr_auc (Precision Recall AUC), logloss (Log Loss), f1, f2, f05 (F-beta scores with beta=1, 2, or 0.5), max_pce, mean_pce (Max/Mean Per-Class Error).
     multiclass: ['logloss', 'acc', 'balacc']        # available metrics: same as for binary, except auc, replaced by auc_ovo (AUC One-vs-One), auc_ovr (AUC One-vs-Rest). AUC metrics and F-beta metrics are computed with weighted average.
     regression: ['rmse', 'r2', 'mae']               # available metrics: mae (Mean Absolute Error), mse (Mean Squared Error), msle (Mean Squared Logarithmic Error), rmse (Root Mean Square Error), rmsle (Root Mean Square Logarithmic Error), r2 (R^2).
+    timeseries: ['mase', 'mape', 'smape', 'rmse', 'mse', 'nrmse', 'wape', 'ncrps']
   defaults:            # the default constraints, usually overridden by a constraint.
     folds: 10          # the amount of fold-runs executed for each dataset.
     max_runtime_seconds: 3600   # default time allocated to the framework to train a model.
@@ -79,7 +80,17 @@ monitoring:               # configuration namespace describing the basic monitor
 
 results:                 # configuration namespace for the results.csv file.
   error_max_length: 200  # the max length of the error message as rendered in the results file.
-  save: true             # set by runbenchmark.py
+  global_save: true      # set by runbenchmark.py, if true adds the results to the main `results.csv` in the {output.dir}
+  global_lock_timeout: 5 # the timeout used to wait for the lock on the global results file.
+  incremental_save: true # if true save results after each job., otherwise save results only when all jobs are completed.
+
+inference_time_measurements:  # configuration namespace for performing additional inference time measurements on various batch sizes
+  enabled: false
+  batch_sizes: [1, 10, 100, 1000, 10000]  # the batch sizes for which inference speed should be measured
+  repeats: 10                            # the number of times to repeat the inference measurement for each batch size
+  additional_job_time: 300  # the time in seconds that will be added to the maximum job time if inference time is measured
+  limit_by_dataset_size: true  # Don't measure inference time on `batch size` if it exceeds the number of rows in the dataset.
+                               # E.g., on micro-mass (571 rows) with `batch_sizes` [1, 10, 100, 1000, 10000], only measure [1, 10, 100].
 
 openml:                # configuration namespace for openML.
   apikey: c1994bdb7ecb3c6f3c8f3b35f4b47f1f
@@ -87,7 +98,7 @@ openml:                # configuration namespace for openML.
 
 versions:              # configuration namespace for versions enforcement (libraries versions are usually enforced in requirements.txt for the app and for each framework).
   pip:
-  python: 3.7          # the Python minor version that will be used by the application in containers and cloud instances, , also used as a based version for virtual environments created for each framework.
+  python: 3.9          # the Python minor version that will be used by the application in containers and cloud instances, also used as a based version for virtual environments created for each framework.
 
 container: &container          # parent configuration namespace for container modes.
   force_branch: true           # set to true if image can only be built from a clean branch, with same tag as defined in `project_repository`.
@@ -103,6 +114,7 @@ container: &container          # parent configuration namespace for container mo
 docker:                        # configuration namespace for docker: it inherits from `container` namespace.
   <<: *container
   run_extra_options: '--shm-size=2048M'
+  build_extra_options: ''
 
 singularity:                   # configuration namespace for docker: it inherits from `container` namespace.
   <<: *container
@@ -137,17 +149,20 @@ aws:                    # configuration namespace for AWS mode.
   ec2:
     regions:                             #
       us-east-1:
-        ami: ami-0ac019f4fcb7cb7e6
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-053b0d53c279acc90
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
+      us-east-2:
+        ami: ami-024e6efaf93d85776
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       us-west-1:
-        ami: ami-063aa838bd7631e0b
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-0f8e81a3da6e2510a
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       eu-west-1:
-        ami: ami-00035f41c82244dab
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-01dd271720c1ba44f
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       eu-central-1:
-        ami: ami-0bdf93799014acdc4
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-04e601abe3e1a910f
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
     instance_type:                       #
       series: m5                         #
       map:                               # map between num cores required and ec2 instance type sizes
@@ -157,7 +172,9 @@ aws:                    # configuration namespace for AWS mode.
         '4': xlarge
         '8': 2xlarge
         '16': 4xlarge
+    instance_tags: {}                    # specify custom tags for the EC2 instances here
     volume_type: standard                # one of gp2, io1, st1, sc1, or standard (default).
+    volume_tags: {}                      # specify custom tags for the volume tags here (if empty, will apply the same tags as the corresponding instance)
     root_device_name: '/dev/sda1'        #
     availability_zone:                   # the availability zone where the instances will be created (if not set, aws will pick a default one).
     subnet_id: ''                        #
diff --git a/resources/frameworks.yaml b/resources/frameworks.yaml
index eb3b7d9e4..74eeee805 100644
--- a/resources/frameworks.yaml
+++ b/resources/frameworks.yaml
@@ -32,9 +32,26 @@ AutoGluon_bestquality:
   extends: AutoGluon
   description: |
     AutoGluon with 'best_quality' preset provides the most accurate overall predictor.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  description: |
+    AutoGluon with 'high_quality' preset provides a very strong predictor with 8x+ faster inference speed than 'best_quality'.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  description: |
+    AutoGluon with 'good_quality' preset provides a strong predictor with 16x+ faster inference speed than 'best_quality'.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'stable'
   description: |
@@ -60,7 +77,8 @@ AutoWEKA:
   refs: [https://www.cs.ubc.ca/labs/beta/Projects/autoweka/papers/16-599.pdf]
 
 autoxgboost:
-  version: 'latest'
+  removed: |
+    Autoxgboost is no longer supported. The last confirmed working version was at v2.0.2 with docker image 'automlbenchmark/autoxgboost:908631665f4c763d548ed203d48f05fe68613844-v2.0.2'.
   description: |
     autoxgboost aims to find an optimal xgboost model automatically using the machine learning framework mlr and the bayesian optimization framework mlrMBO.
   project: https://github.com/ja-thomas/autoxgboost
@@ -69,9 +87,9 @@ autoxgboost:
 flaml:
   version: 'stable'
   description: |
-    FLAML is a lightweight Python library that finds accurate machine learning models 
-    automatically, efficiently and economically. It frees users from selecting learners 
-    and hyperparameters for each learner. It is fast and cheap. 
+    FLAML is a lightweight Python library that finds accurate machine learning models
+    automatically, efficiently and economically. It frees users from selecting learners
+    and hyperparameters for each learner. It is fast and cheap.
   project: https://github.com/microsoft/FLAML
   refs: [https://arxiv.org/pdf/1911.04706.pdf]
 
@@ -122,12 +140,12 @@ mljarsupervised_compete:
   description: "MLJAR is using 'Compete' mode to provide the most accurate predictor"
   params:
     mode: Compete   # set mode for Compete, default mode is Explain
-    
+
 MLNet:
   version: 'latest'
   description: |
     MLNET.CLI is a automated machine learning tool implemented by ml.net.
-    
+
 MLPlan:
   version: 'stable'
   abstract: true
@@ -174,7 +192,17 @@ TPOT:
 #    population_size: 25
 #    verbosity: 2
 
+####################################
+### TimeSeries AutoML frameworks ###
+####################################
 
+AutoGluonTS:
+  extends: AutoGluon
+  version: "stable"
+  description: |
+    AutoGluon-TimeSeries
+  setup_env:
+    MODULE: timeseries
 
 #######################################
 ### Non AutoML reference frameworks ###
@@ -217,4 +245,3 @@ TunedRandomForest:
 #    _n_jobs: 1  # cf. RandomForest
 #    _tuning:
 #      n_estimators: 500
-
diff --git a/resources/frameworks_2020Q2.yaml b/resources/frameworks_2020Q2.yaml
index 218827276..b8ace6c21 100644
--- a/resources/frameworks_2020Q2.yaml
+++ b/resources/frameworks_2020Q2.yaml
@@ -19,9 +19,6 @@ autosklearn:
 AutoWEKA:
   version: '2.6'
 
-autoxgboost:
-  version: 'latest'
-
 GAMA:
   version: '20.1.0'
 
diff --git a/resources/frameworks_2021Q3.yaml b/resources/frameworks_2021Q3.yaml
index cea36f5ba..ad1b42e8e 100644
--- a/resources/frameworks_2021Q3.yaml
+++ b/resources/frameworks_2021Q3.yaml
@@ -22,9 +22,6 @@ autosklearn2:
   params:
     _askl2: true
 
-autoxgboost:
-  version: '#908631665f4c763d548ed203d48f05fe68613844'
-
 AutoWEKA:
   version: '2.6'
 
diff --git a/resources/frameworks_2023Q2.yaml b/resources/frameworks_2023Q2.yaml
new file mode 100644
index 000000000..2eff34321
--- /dev/null
+++ b/resources/frameworks_2023Q2.yaml
@@ -0,0 +1,119 @@
+---
+
+#########################
+### AutoML frameworks ###
+#########################
+
+AutoGluon:
+  abstract: true
+  version: "0.8.0"
+
+AutoGluon_benchmark:
+  extends: AutoGluon
+  description: "AutoGluon with 'best_quality' preset provides the most accurate overall predictor"
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: best_quality
+    _leaderboard_test: True
+
+AutoGluon_hq:
+  extends: AutoGluon
+  description: "AutoGluon with 'high_quality' preset provides generally fast inference speed with high accuracy"
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: high_quality
+    _leaderboard_test: True
+
+AutoGluon_hq_il001:
+  extends: AutoGluon
+  description: "AutoGluon ~3x faster inference at slight performance loss to 'high quality' (self-reported)."
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: high_quality
+    _leaderboard_test: True
+    infer_limit: 0.01
+
+autosklearn:
+  version: '0.15.0'
+
+autosklearn2:
+  extends: autosklearn
+  params:
+    _askl2: true
+
+AutoWEKA:
+  version: '2.6'
+
+flaml:
+  version: '1.2.4'
+
+GAMA:
+  abstract: true
+  version: '22.0.0'
+
+GAMA_benchmark:
+  extends: GAMA
+  params:
+    goal: performance
+
+H2OAutoML:
+  version: '3.40.0.4'
+
+lightautoml:
+  version: '0.3.7.3'
+
+mljarsupervised:
+  abstract: true
+  version: '0.11.5'
+
+mljarsupervised_benchmark:
+  extends: mljarsupervised
+  description: "MLJar using 'Compete' mode to provide most accurate predictor"
+  params:
+    mode: Compete
+
+mljarsupervised_perform:
+  extends: mljarsupervised
+  description: "MLJar using 'Perform' mode to provide fast inference"
+  params:
+    mode: Perform
+
+MLPlan:
+  abstract: true
+  version: '0.2.5'
+
+MLPlanWEKA:
+  extends: MLPlan
+  params:
+    _backend: weka
+
+mlr3automl:
+  version: '#f667900292e3ded64bb419285c71cd5d1d2c4301'
+  project: https://github.com/a-hanf/mlr3automl
+
+TPOT:
+  version: '0.12.0'
+
+
+#######################################
+### Non AutoML reference frameworks ###
+#######################################
+
+constantpredictor: {}
+
+DecisionTree:
+  version: '1.2.2'
+
+RandomForest:
+  version: '1.2.2'
+  params:
+    n_estimators: 2000
+
+ranger:
+  version: 'stable'
+
+TunedRandomForest:
+  version: '1.2.2'
+  params:
+    n_estimators: 2000
+
diff --git a/resources/frameworks_stable.yaml b/resources/frameworks_stable.yaml
index ff55160bf..15c6deb33 100644
--- a/resources/frameworks_stable.yaml
+++ b/resources/frameworks_stable.yaml
@@ -15,6 +15,16 @@ AutoGluon_bestquality:
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'stable'
 
@@ -26,15 +36,24 @@ autosklearn2:
 AutoWEKA:
   version: 'stable'
 
-autoxgboost:
+
+flaml:
   version: 'stable'
 
+
+flaml_benchmark:
+  extends: flaml
+  version: "benchmark"
+
 GAMA:
   version: 'stable'
 
 H2OAutoML:
   version: 'stable'
 
+lightautoml:
+  version: 'stable'
+
 hyperoptsklearn:
   version: 'stable'
 
@@ -60,6 +79,9 @@ MLPlanWEKA:
   params:
     _backend: weka
 
+mlr3automl:
+  version: 'stable'
+
 oboe:
   version: 'stable'
 
diff --git a/runbenchmark.py b/runbenchmark.py
index 7ff8c6065..ad5eac7e3 100644
--- a/runbenchmark.py
+++ b/runbenchmark.py
@@ -8,19 +8,12 @@
 # prevent asap other modules from defining the root logger using basicConfig
 import amlb.logger
 
-from openml.config import cache_directory
+import openml
 
 import amlb
 from amlb.utils import Namespace as ns, config_load, datetime_iso, str2bool, str_sanitize, zip_path
 from amlb import log, AutoMLError
-
-
-default_dirs = ns(
-    input_dir=cache_directory,
-    output_dir="./results",
-    user_dir="~/.config/automlbenchmark",
-    root_dir=os.path.dirname(__file__)
-)
+from amlb.defaults import default_dirs
 
 parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
 parser.add_argument('framework', type=str,
@@ -31,7 +24,8 @@
                     help="The benchmark type to run as defined by default in resources/benchmarks/{benchmark}.yaml,"
                          "\na path to a benchmark description file, or an openml suite or task."
                          "\nOpenML references should be formatted as 'openml/s/X' and 'openml/t/Y',"
-                         "\nfor studies and tasks respectively."
+                         "\nfor studies and tasks respectively. Use 'test.openml/s/X' for the "
+                         "\nOpenML test server."
                          "\n(default: '%(default)s')")
 parser.add_argument('constraint', type=str, nargs='?', default='test',
                     help="The constraint definition to use as defined by default in resources/constraints.yaml."
@@ -84,6 +78,11 @@
                          "\n  --logging=root:debug (keeps defaults for non-specified loggers)" 
                          "\n  --logging=console:warning,app:info"
                          "\n(default: '%(default)s')")
+parser.add_argument('--openml-test-server', type=str2bool, metavar='true|false', nargs='?', const=True, default=False,
+                    help=argparse.SUPPRESS)  # "Set to true to connect to the OpenML test server instead."
+parser.add_argument('--openml-run-tag', type=str, default=None,
+                    help="Tag that will be saved in metadata and OpenML runs created during upload, must match '([a-zA-Z0-9_\-\.])+'.")
+
 parser.add_argument('--profiling', nargs='?', const=True, default=False, help=argparse.SUPPRESS)
 parser.add_argument('--resume', nargs='?', const=True, default=False, help=argparse.SUPPRESS)
 parser.add_argument('--session', type=str, default=None, help=argparse.SUPPRESS)
@@ -119,7 +118,7 @@
 # now_str = datetime_iso(time=False, no_sep=True)
 if args.profiling:
     logging.TRACE = logging.INFO
-log_levels = ns({logger: level.upper()
+log_levels = ns({logger: int(level) if level.isnumeric() else level.upper()
                  for logger, level in [d.split(':') for d in args.logging.split(',')]} if ':' in args.logging
                 else dict(console=args.logging.upper(), app=args.logging.upper(), root=args.logging.upper()) if args.logging
                 else {}) | ns(console='INFO', app='DEBUG', root='INFO')  # adding defaults if needed
@@ -128,6 +127,10 @@
                   root_level=log_levels.root, app_level=log_levels.app, console_level=log_levels.console, print_to_log=True)
 
 log.info("Running benchmark `%s` on `%s` framework in `%s` mode.", args.framework, args.benchmark, args.mode)
+if args.openml_test_server:
+    openml.config.start_using_configuration_for_example()
+    log.info("Connecting to the OpenML test server.")
+
 log.debug("Script args: %s.", args)
 
 config_default = config_load(os.path.join(default_dirs.root_dir, "resources", "config.yaml"))
@@ -136,7 +139,7 @@
 config_user = config_load(extras.get('config', os.path.join(args.userdir or default_dirs.user_dir, "config.yaml")))
 # config listing properties set by command line
 config_args = ns.parse(
-    {'results.save': args.keep_scores},
+    {'results.global_save': args.keep_scores},
     input_dir=args.indir,
     output_dir=args.outdir,
     user_dir=args.userdir,
@@ -145,6 +148,9 @@
     parallel_jobs=args.parallel,
     sid=sid,
     exit_on_error=args.exit_on_error,
+    test_server=args.openml_test_server,
+    tag=args.openml_run_tag,
+    command=' '.join(sys.argv),
 ) + ns.parse(extras)
 if args.mode != 'local':
     config_args + ns.parse({'monitoring.frequency_seconds': 0})
@@ -155,8 +161,8 @@
 if args.resume:
     amlb_res.config.job_history = os.path.join(amlb_res.config.output_dir, amlb.results.Scoreboard.results_file)
 
-code = 0
 bench = None
+exit_code = 0
 try:
     if args.mode == 'local':
         bench = amlb.Benchmark(args.framework, args.benchmark, args.constraint)
@@ -185,10 +191,10 @@
     log.error('\nERROR:\n%s', e)
     if extras.get('verbose') is True:
         log.exception(e)
-    code = 1
+    exit_code = 1
 except Exception as e:
     log.exception(e)
-    code = 2
+    exit_code = 2
 finally:
     archives = amlb.resources.config().archive
     if archives and bench:
@@ -197,5 +203,7 @@
             if d in out_dirs:
                 zip_path(out_dirs[d], os.path.join(out_dirs.session, f"{d}.zip"), arc_path_format='long')
                 shutil.rmtree(out_dirs[d], ignore_errors=True)
+    if args.openml_test_server:
+        openml.config.stop_using_configuration_for_example()
 
-    sys.exit(code)
+    sys.exit(exit_code)
diff --git a/runstable.sh b/runstable.sh
index 28d8628f9..3be2857e6 100755
--- a/runstable.sh
+++ b/runstable.sh
@@ -12,7 +12,6 @@ h2oautoml
 tpot
 oboe
 autoweka
-autoxgboost
 #hyperoptsklearn
 #ranger
 )
diff --git a/upload_results.py b/upload_results.py
new file mode 100644
index 000000000..800cde0f7
--- /dev/null
+++ b/upload_results.py
@@ -0,0 +1,188 @@
+""" Script to upload results from the benchmark to OpenML.
+The benchmark run needs to be executed on OpenML datasets to be eligible for upload.
+"""
+import argparse
+import contextlib
+from contextlib import contextmanager
+from datetime import datetime
+import logging
+import os
+import pathlib
+from typing import Optional
+
+import openml
+from openml import OpenMLRun
+
+from amlb.defaults import default_dirs
+from amlb.resources import config_load
+from amlb.uploads import process_task_folder, missing_folds, _load_task_data
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.DEBUG)
+
+
+def parse_args():
+    description = "Script to upload results from the benchmark to OpenML."
+    parser = argparse.ArgumentParser(description)
+    parser.add_argument(
+        '-i', '--input-directory', type=pathlib.Path, default=None, dest='input_directory',
+        help="Directory that stores results from the runbenchmark.py invocation. "
+             "By default use the most recent folder in the results "
+             "folder as specified in the configuration."
+    )
+    parser.add_argument(
+        '-a', '--api-key', dest='apikey', default=None,
+        help="By default, the api key configured in your OpenML configuration file is"
+             "used. Specify this parameter if you want to overwrite this value or"
+             "you do not have an OpenML configuration file. A valid key is "
+             "*required* to upload to the OpenML server."
+             "This argument is ignored when uploading to the test server, as "
+             "the default openml-python test server api key will be used instead."
+    )
+    parser.add_argument(
+        '-m', '--mode', dest='mode', default='check',
+        help="Run mode (default=%(default)s)."
+             "• check: only report whether results can be uploaded."
+             "• upload: upload all complete results."
+    )
+    parser.add_argument(
+        '-x', '--fail-fast', dest='fail_fast', action='store_true',
+        help="Stop as soon as a task fails to upload due to an error during uploading."
+    )
+    parser.add_argument(
+        '-v', '--verbose', action='store_true', dest='verbose',
+        help="Output progress to console."
+    )
+    parser.add_argument(
+        '-t', '--task', type=str, dest='task', default=None,
+        help="Only upload results for this specific task."
+    )
+    args = parser.parse_args()
+
+    if args.mode not in ['check', 'upload']:
+        raise ValueError(f"Invalid value for argument 'mode': '{args.mode}'.")
+
+    return args
+
+
+def find_most_recent_result_folder() -> pathlib.Path:
+    root_dir = pathlib.Path(__file__).parent
+    config = config_load(root_dir / "resources" / "config.yaml")
+    output_dir = pathlib.Path(config.output_dir or default_dirs.output_dir)
+
+    def dirname_to_datetime(dirname: str) -> datetime:
+        _, timestamp = dirname.rsplit('.', 1)
+        return datetime.strptime(timestamp, "%Y%m%dT%H%M%S")
+
+    run_directories = output_dir.glob("*.*.*.*")
+    _, run_directory = max((dirname_to_datetime(str(d)), d) for d in run_directories)
+    return run_directory
+
+
+def resolve_input_directory(path: Optional[pathlib.Path]) -> pathlib.Path:
+    path = path or find_most_recent_result_folder()
+    path = path.expanduser().absolute()
+    if not path.is_dir():
+        raise ValueError(f"{path} is not a directory.")
+    return path
+
+
+def configure_logging(verbose: bool):
+    formatter = logging.Formatter(
+        '[%(asctime)s] %(levelname)s: %(message)s',
+        '%H:%M:%S',
+    )
+    log_level = logging.DEBUG if verbose else logging.INFO
+
+    console = logging.StreamHandler()
+    console.setFormatter(formatter)
+    console.setLevel(log_level)
+
+    log.addHandler(console)
+
+
+@contextmanager
+def _connect_to_test_server():
+    openml.config.start_using_configuration_for_example()
+    yield
+    openml.config.stop_using_configuration_for_example()
+
+
+def server_for_task(task: pathlib.Path):
+    metadata = _load_task_data(task)
+    if metadata.test_server:
+        server_connection = _connect_to_test_server()
+    else:
+        server_connection = contextlib.nullcontext()
+    return server_connection
+
+
+def upload_task(task_directory: pathlib.Path) -> Optional[OpenMLRun]:
+    task_name = os.path.basename(task_directory)
+    try:
+        with server_for_task(task_directory):
+            log.debug("Starting upload for '%s'." % task_name)
+            run = process_task_folder(task_directory)
+            log.info("%s result stored at %s/r/%d"
+                     % (task_name, openml.config.server[:-11], run.id))
+            return run
+    except Exception as e:
+        message = e.message if hasattr(e, "message") else e.args[0]
+        log.warning("Task %s failed to upload: %s" % (task_name, message))
+        if args.fail_fast:
+            raise
+
+
+def process_results(result_dir: pathlib.Path, mode: str = 'check'):
+    prediction_directory = result_dir / "predictions"
+    if not prediction_directory.exists():
+        raise ValueError(f"result_dir '{result_dir!s}' has no predictions.")
+
+    if args.task is None:
+        tasks_to_process = [d for d in prediction_directory.iterdir() if d.is_dir()]
+    elif (prediction_directory / args.task).is_dir():
+        tasks_to_process = [args.task]
+    else:
+        raise ValueError(f"Task '{args.task}' not in '{prediction_directory}'.")
+
+    for task_name in tasks_to_process:
+        full_task_directory = prediction_directory / task_name
+
+        folds = missing_folds(full_task_directory)
+        if len(folds) > 0:
+            log.info("%s has missing folds: %s" % (task_name, ', '.join(sorted(folds))))
+            continue
+
+        metadata = _load_task_data(full_task_directory)
+        if "openml_task_id" not in metadata:
+            log.info("%s has openml task metadata" % task_name)
+            continue
+
+        if mode == 'check':
+            log.info("%s is ready for upload." % task_name)
+        elif mode == 'upload':
+            upload_task(full_task_directory)
+
+
+def configure_apikey(key: Optional[str]) -> bool:
+    if key:
+        openml.config.apikey = key
+    # API does not support easy checking of validity
+    is_valid = True
+    return openml.config.apikey is not None and is_valid
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    configure_logging(args.verbose)
+    valid_key = configure_apikey(args.apikey)
+    if not valid_key and args.mode == 'upload':
+        raise ValueError(
+            "No valid OpenML API key configured, use the '--api-key' argument "
+            "or follow instructions: https://openml.github.io/openml-python/master/usage.html#configuration"
+        )
+    input_directory = resolve_input_directory(args.input_directory)
+    mode_verb = 'Uploading' if args.mode == 'upload' else 'Checking'
+    log.info("%s results from '%s'." % (mode_verb, input_directory))
+
+    process_results(input_directory, args.mode)


Test patch:
diff --git a/.github/workflows/run_pytest.yaml b/.github/workflows/run_pytest.yaml
new file mode 100644
index 000000000..52be3c836
--- /dev/null
+++ b/.github/workflows/run_pytest.yaml
@@ -0,0 +1,35 @@
+name: Light-weight Unit Tests
+
+on: pull_request
+
+
+jobs:
+  pytest_not_stress:
+    name: pytest
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+      - name: Create venv
+        run: python -m venv venv
+      - uses: actions/cache@v3
+        id: cache
+        with:
+          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
+          key: pip-v3-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/requirements-dev.txt') }}
+          restore-keys: |
+            pip-v3-
+      - name: Install Requirements
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: |
+          source venv/bin/activate
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+      - name: Run unit tests
+        run: |
+          source venv/bin/activate
+          python -m pytest -vv -m "not stress and not openml_upload"
diff --git a/resources/frameworks_latest.yaml b/resources/frameworks_latest.yaml
index 0a4705d96..adb90f360 100644
--- a/resources/frameworks_latest.yaml
+++ b/resources/frameworks_latest.yaml
@@ -16,6 +16,16 @@ AutoGluon_bestquality:
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'latest'
 
@@ -27,9 +37,6 @@ autosklearn2:
 AutoWEKA:
   version: 'latest'
 
-autoxgboost:
-  version: 'latest'
-
 GAMA:
   version: 'latest'
 
@@ -50,6 +57,9 @@ mljarsupervised_compete:
   params:
     mode: Compete   # set mode for Compete, default mode is Explain
 
+MLNet:
+  version: 'latest'
+
 MLPlan:
   abstract: true
   version: 'latest'
@@ -64,14 +74,26 @@ MLPlanWEKA:
   params:
     _backend: weka
 
+mlr3automl:
+  version: 'latest'
+
 oboe:
   version: 'latest'
 
 TPOT:
   version: 'latest'
 
+####################################
+### TimeSeries AutoML frameworks ###
+####################################
 
-
+AutoGluonTS:
+  extends: AutoGluon
+  version: "latest"
+  description: |
+    AutoGluon-TimeSeries
+  setup_env:
+    MODULE: timeseries
 
 #######################################
 ### Non AutoML reference frameworks ###
@@ -94,4 +116,3 @@ TunedRandomForest:
   version: 'latest'
   params:
     n_estimators: 2000
-
diff --git a/tests/unit/amlb/datasets/file/resources/iris_num_test.csv b/tests/unit/amlb/datasets/file/resources/iris_num_test.csv
new file mode 100644
index 000000000..af6e38257
--- /dev/null
+++ b/tests/unit/amlb/datasets/file/resources/iris_num_test.csv
@@ -0,0 +1,16 @@
+sepallength,sepalwidth,petallength,petalwidth,class
+5.0,3.5,1.6,0.6,1
+5.8,4.0,1.2,0.2,1
+4.9,3.1,1.5,0.1,1
+5.1,3.3,1.7,0.5,1
+5.4,3.7,1.5,0.2,1
+5.7,2.8,4.1,1.3,2
+6.3,2.3,4.4,1.3,2
+6.2,2.9,4.3,1.3,2
+6.0,2.2,4.0,1.0,2
+5.8,2.6,4.0,1.2,2
+6.0,2.2,5.0,1.5,3
+6.4,2.7,5.3,1.9,3
+6.7,3.3,5.7,2.5,3
+6.5,3.0,5.5,1.8,3
+7.2,3.2,6.0,1.8,3
diff --git a/tests/unit/amlb/datasets/file/resources/iris_num_train.csv b/tests/unit/amlb/datasets/file/resources/iris_num_train.csv
new file mode 100644
index 000000000..17c83f6f1
--- /dev/null
+++ b/tests/unit/amlb/datasets/file/resources/iris_num_train.csv
@@ -0,0 +1,136 @@
+sepallength,sepalwidth,petallength,petalwidth,class
+5.0,3.3,1.4,0.2,1
+5.7,4.4,1.5,0.4,1
+4.6,3.2,1.4,0.2,1
+5.1,3.5,1.4,0.2,1
+5.1,3.8,1.9,0.4,1
+6.8,2.8,4.8,1.4,2
+5.5,2.5,4.0,1.3,2
+6.6,2.9,4.6,1.3,2
+6.5,2.8,4.6,1.5,2
+5.5,2.3,4.0,1.3,2
+5.9,3.0,5.1,1.8,3
+5.8,2.8,5.1,2.4,3
+6.8,3.0,5.5,2.1,3
+6.4,3.2,5.3,2.3,3
+6.9,3.1,5.4,2.1,3
+4.4,3.2,1.3,0.2,1
+4.5,2.3,1.3,0.3,1
+5.4,3.4,1.5,0.4,1
+5.3,3.7,1.5,0.2,1
+5.0,3.4,1.5,0.2,1
+5.6,2.7,4.2,1.3,2
+5.0,2.3,3.3,1.0,2
+5.7,3.0,4.2,1.2,2
+6.4,3.2,4.5,1.5,2
+6.3,2.5,4.9,1.5,2
+7.7,3.0,6.1,2.3,3
+6.7,3.3,5.7,2.1,3
+6.9,3.1,5.1,2.3,3
+7.6,3.0,6.6,2.1,3
+6.8,3.2,5.9,2.3,3
+4.9,3.1,1.5,0.1,1
+4.6,3.6,1.0,0.2,1
+4.8,3.0,1.4,0.3,1
+4.6,3.1,1.5,0.2,1
+5.4,3.9,1.3,0.4,1
+5.7,2.6,3.5,1.0,2
+5.7,2.8,4.5,1.3,2
+5.1,2.5,3.0,1.1,2
+5.5,2.4,3.8,1.1,2
+6.7,3.0,5.0,1.7,2
+6.9,3.2,5.7,2.3,3
+7.7,2.8,6.7,2.0,3
+6.7,3.0,5.2,2.3,3
+6.3,3.3,6.0,2.5,3
+7.3,2.9,6.3,1.8,3
+5.0,3.6,1.4,0.2,1
+5.2,3.5,1.5,0.2,1
+4.4,3.0,1.3,0.2,1
+5.1,3.8,1.5,0.3,1
+5.5,4.2,1.4,0.2,1
+5.7,2.9,4.2,1.3,2
+7.0,3.2,4.7,1.4,2
+5.9,3.0,4.2,1.5,2
+6.7,3.1,4.4,1.4,2
+6.1,2.8,4.7,1.2,2
+7.2,3.6,6.1,2.5,3
+6.2,2.8,4.8,1.8,3
+6.5,3.2,5.1,2.0,3
+6.2,3.4,5.4,2.3,3
+6.3,2.9,5.6,1.8,3
+5.1,3.8,1.6,0.2,1
+4.6,3.4,1.4,0.3,1
+5.7,3.8,1.7,0.3,1
+5.5,3.5,1.3,0.2,1
+4.8,3.4,1.9,0.2,1
+6.0,2.7,5.1,1.6,2
+5.5,2.6,4.4,1.2,2
+6.2,2.2,4.5,1.5,2
+6.9,3.1,4.9,1.5,2
+6.6,3.0,4.4,1.4,2
+6.1,2.6,5.6,1.4,3
+6.5,3.0,5.8,2.2,3
+7.7,3.8,6.7,2.2,3
+6.1,3.0,4.9,1.8,3
+5.6,2.8,4.9,2.0,3
+5.0,3.4,1.6,0.4,1
+4.3,3.0,1.1,0.1,1
+5.2,4.1,1.5,0.1,1
+5.4,3.4,1.7,0.2,1
+5.1,3.5,1.4,0.3,1
+6.1,2.9,4.7,1.4,2
+5.6,3.0,4.1,1.3,2
+5.6,2.9,3.6,1.3,2
+6.1,3.0,4.6,1.4,2
+5.5,2.4,3.7,1.0,2
+6.4,3.1,5.5,1.8,3
+5.8,2.7,5.1,1.9,3
+7.4,2.8,6.1,1.9,3
+7.7,2.6,6.9,2.3,3
+6.7,2.5,5.8,1.8,3
+4.7,3.2,1.6,0.2,1
+5.0,3.5,1.3,0.3,1
+4.8,3.0,1.4,0.1,1
+5.0,3.2,1.2,0.2,1
+4.4,2.9,1.4,0.2,1
+6.7,3.1,4.7,1.5,2
+5.8,2.7,3.9,1.2,2
+5.4,3.0,4.5,1.5,2
+6.4,2.9,4.3,1.3,2
+4.9,2.4,3.3,1.0,2
+6.7,3.1,5.6,2.4,3
+6.3,2.8,5.1,1.5,3
+4.9,2.5,4.5,1.7,3
+6.4,2.8,5.6,2.1,3
+7.1,3.0,5.9,2.1,3
+5.0,3.0,1.6,0.2,1
+4.9,3.0,1.4,0.2,1
+5.1,3.7,1.5,0.4,1
+4.8,3.1,1.6,0.2,1
+5.2,3.4,1.4,0.2,1
+5.8,2.7,4.1,1.0,2
+6.0,2.9,4.5,1.5,2
+5.6,3.0,4.5,1.5,2
+5.9,3.2,4.8,1.8,2
+5.2,2.7,3.9,1.4,2
+6.3,3.4,5.6,2.4,3
+6.0,3.0,4.8,1.8,3
+7.2,3.0,5.8,1.6,3
+5.7,2.5,5.0,2.0,3
+6.4,2.8,5.6,2.2,3
+5.4,3.9,1.7,0.4,1
+4.8,3.4,1.6,0.2,1
+4.9,3.1,1.5,0.1,1
+5.1,3.4,1.5,0.2,1
+4.7,3.2,1.3,0.2,1
+6.0,3.4,4.5,1.6,2
+6.3,3.3,4.7,1.6,2
+5.6,2.5,3.9,1.1,2
+6.1,2.8,4.0,1.3,2
+5.0,2.0,3.5,1.0,2
+5.8,2.7,5.1,1.9,3
+6.3,2.7,4.9,1.8,3
+7.9,3.8,6.4,2.0,3
+6.5,3.0,5.2,2.0,3
+6.3,2.5,5.0,1.9,3
diff --git a/tests/unit/amlb/datasets/file/test_file_dataloader.py b/tests/unit/amlb/datasets/file/test_file_dataloader.py
index b979a78cc..fa8151789 100644
--- a/tests/unit/amlb/datasets/file/test_file_dataloader.py
+++ b/tests/unit/amlb/datasets/file/test_file_dataloader.py
@@ -65,7 +65,8 @@ def test_load_binary_task_arff(file_loader):
 def _assert_kc2_features(dataset, definition):
     assert len(dataset.features) == 22
     assert len(dataset.predictors) == 21
-    _assert_target(dataset.target, name=definition.target, values=["no", "yes"])
+    target_values = ["no", "yes"]
+    _assert_target(dataset.target, name=definition.target, values=target_values)
 
     assert all([p.data_type in ['int', 'float'] for p in dataset.predictors])
     assert all([p.values is None for p in dataset.predictors])
@@ -78,6 +79,10 @@ def _assert_kc2_features(dataset, definition):
     assert dataset.train.X.dtypes.filter(items=ints).apply(lambda dt: pd.api.types.is_integer_dtype(dt)).all()
     assert pd.api.types.is_categorical_dtype(dataset.train.y.dtypes.iloc[0])
 
+    normalize = dataset.target.normalize
+    assert list(normalize(dataset.train.y.squeeze().unique())) == list(normalize(dataset.test.y.squeeze().unique())) == target_values
+    assert list(np.unique(dataset.train.y_enc)) == list(np.unique(dataset.test.y_enc)) == [0, 1]
+
 
 @pytest.mark.use_disk
 def test_load_multiclass_task_csv(file_loader):
@@ -94,6 +99,33 @@ def test_load_multiclass_task_csv(file_loader):
     _assert_iris_features(ds, ds_def)
 
 
+@pytest.mark.use_disk
+def test_load_multiclass_task_with_num_target_no_type_csv(file_loader):
+    ds_def = ns(
+        train=os.path.join(res, "iris_num_train.csv"),
+        test=os.path.join(res, "iris_num_test.csv"),
+        target="class"
+    )
+    ds = file_loader.load(ds_def)
+    assert ds.type is DatasetType.regression, "file loader should detect num target as regression by default"
+
+
+@pytest.mark.use_disk
+def test_load_multiclass_task_with_num_target_csv(file_loader):
+    ds_def = ns(
+        train=os.path.join(res, "iris_num_train.csv"),
+        test=os.path.join(res, "iris_num_test.csv"),
+        target="class",
+        type="multiclass"
+    )
+    ds = file_loader.load(ds_def)
+    assert ds.type is DatasetType.multiclass
+    _assert_X_y_types(ds.train)
+    _assert_data_consistency(ds)
+    _assert_data_paths(ds, ds_def)
+    _assert_iris_features(ds, ds_def, num_target=True)
+
+
 @pytest.mark.use_disk
 def test_load_multiclass_task_arff(file_loader):
     ds_def = ns(
@@ -109,10 +141,11 @@ def test_load_multiclass_task_arff(file_loader):
     _assert_iris_features(ds, ds_def)
 
 
-def _assert_iris_features(dataset, definition):
+def _assert_iris_features(dataset, definition, num_target=False):
     assert len(dataset.features) == 5
     assert len(dataset.predictors) == 4
-    _assert_target(dataset.target, name=definition.target, values=["Iris-setosa", "Iris-versicolor", "Iris-virginica"])  # values are case-sensitive when using file loader
+    target_values = ["1", "2", "3"] if num_target else ["iris-setosa", "iris-versicolor", "iris-virginica"]  # values are normalized
+    _assert_target(dataset.target, name=definition.target, values=target_values)
 
     assert all([p.data_type in ['int', 'float'] for p in dataset.predictors])
     assert all([p.values is None for p in dataset.predictors])
@@ -125,6 +158,10 @@ def _assert_iris_features(dataset, definition):
     assert dataset.train.X.dtypes.filter(items=ints).apply(lambda dt: pd.api.types.is_integer_dtype(dt)).all()
     assert pd.api.types.is_categorical_dtype(dataset.train.y.dtypes.iloc[0])
 
+    normalize = dataset.target.normalize
+    assert list(normalize(dataset.train.y.squeeze().unique())) == list(normalize(dataset.test.y.squeeze().unique())) == target_values
+    assert list(np.unique(dataset.train.y_enc)) == list(np.unique(dataset.test.y_enc)) == [0, 1, 2]
+
 
 @pytest.mark.use_disk
 def test_load_regression_task_csv(file_loader):
@@ -178,6 +215,9 @@ def _assert_cholesterol_features(dataset, definition, fmt):
     assert dataset.train.X.dtypes.filter(items=categoricals).apply(lambda dt: pd.api.types.is_categorical_dtype(dt)).all()
     assert pd.api.types.is_float_dtype(dataset.train.y.dtypes.iloc[0])
 
+    assert np.array_equal(dataset.train.y_enc, dataset.train.y.squeeze().to_numpy()), "no encoding should have been applied on regression target"
+    assert np.array_equal(dataset.test.y_enc, dataset.test.y.squeeze().to_numpy()), "no encoding should have been applied on regression target"
+
 
 def _assert_target(target, name, values=None):
     assert target.name == name
diff --git a/tests/unit/amlb/datasets/openml/test_openml_dataloader.py b/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
index 8b1efbb35..3c496d462 100644
--- a/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
+++ b/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
@@ -113,8 +113,7 @@ def _assert_cholesterol_features(dataset):
 
     assert dataset.train.X.dtypes.filter(items=numericals).apply(lambda dt: pd.api.types.is_numeric_dtype(dt)).all()
     assert dataset.train.X.dtypes.filter(items=categoricals).apply(lambda dt: pd.api.types.is_categorical_dtype(dt)).all()
-    assert len(dataset.train.X.select_dtypes(include=['float']).columns) == 2
-    assert len(dataset.train.X.select_dtypes(include=['uint8']).columns) == 4
+    assert len(dataset.train.X.select_dtypes(include=['float']).columns) == 6
     assert pd.api.types.is_float_dtype(dataset.train.y.dtypes.iloc[0])
 
 
diff --git a/tests/unit/amlb/job/test_MultiThreadingJobRunner.py b/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
index 543c2285c..7510f1906 100644
--- a/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
+++ b/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
@@ -124,6 +124,7 @@ def test_reschedule_job_enforce_job_priority():
 
     runner = MultiThreadingJobRunner(jobs, parallel_jobs=3, delay_secs=0.2,
                                      queueing_strategy=MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority)
+
     def _run(self, mock, ori):
         if mock.call_count < 3:
             self.reschedule()
@@ -228,9 +229,9 @@ def _on_state(self, state, ori=None):
 
     rescheduled_jobs = [j for i, j in enumerate(jobs) if i % 5 == 0]
     for job in rescheduled_jobs:
-        job._run = ft.partialmethod(_run, ori=job._run).__get__(job)
+        job._run = ft.partialmethod(_run, ori=job._run).__get__(job, None)
     for job in jobs:
-        job._on_state = ft.partialmethod(_on_state, ori=job._on_state).__get__(job)
+        job._on_state = ft.partialmethod(_on_state, ori=job._on_state).__get__(job, None)
 
     runner = MultiThreadingJobRunner(jobs, parallel_jobs=50, delay_secs=0.01,
                                      queueing_strategy=MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority)
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json
new file mode 100644
index 000000000..4ad103514
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 0,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575752,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/0/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/0/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json
new file mode 100644
index 000000000..95be8717b
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 1,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575753,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/1/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/1/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json
new file mode 100644
index 000000000..b18f172c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 2,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575754,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/2/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/2/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json
new file mode 100644
index 000000000..e9ebbfc65
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 3,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575755,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/3/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/3/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json
new file mode 100644
index 000000000..06ebf6b37
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 4,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575756,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/4/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/4/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json
new file mode 100644
index 000000000..caf373827
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 5,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575757,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/5/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/5/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json
new file mode 100644
index 000000000..94e735223
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 6,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575758,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2461,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/6/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/6/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json
new file mode 100644
index 000000000..83af8f9bd
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 7,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575759,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2463,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/7/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/7/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json
new file mode 100644
index 000000000..ee7d85454
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 8,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575760,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2462,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/8/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/8/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json
new file mode 100644
index 000000000..16dd4df8c
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 9,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575761,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2460,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/9/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/9/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json
new file mode 100644
index 000000000..7711c2c73
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 0,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415600,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22324,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/0/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/0/metadata.json",
+    "result_token": "0e96514e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp6q9_eifx",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv
new file mode 100644
index 000000000..26e5202af
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9946953997563966,0.0053046002344551485,9.148348333343108E-12,Iris-setosa,Iris-setosa
+0.9985317282709597,0.0014682717290367799,3.6521815928739156E-15,Iris-setosa,Iris-setosa
+0.9933721446687596,0.006627855330845209,3.952500544573547E-13,Iris-setosa,Iris-setosa
+0.9829199191390252,0.017080080834738047,2.623664472860213E-11,Iris-setosa,Iris-setosa
+0.9975765427390224,0.00242345726092685,5.062201017884491E-14,Iris-setosa,Iris-setosa
+0.0044465533046761044,0.9892091660989077,0.006344280596416092,Iris-versicolor,Iris-versicolor
+3.514690516726789E-5,0.9714925562042298,0.028472296890602915,Iris-versicolor,Iris-versicolor
+0.0010697343341420122,0.992866262840173,0.006064002825685055,Iris-versicolor,Iris-versicolor
+2.1266769099902806E-4,0.9986645376592181,0.0011227946497828257,Iris-versicolor,Iris-versicolor
+0.001757984985526363,0.9954098065054922,0.002832208508981477,Iris-versicolor,Iris-versicolor
+5.742239231322204E-6,0.33211727254434054,0.6678769852164281,Iris-virginica,Iris-virginica
+4.121706214335948E-7,0.031571169758323414,0.9684284180710553,Iris-virginica,Iris-virginica
+4.066267276715623E-9,3.412176268706058E-4,0.9996587783068621,Iris-virginica,Iris-virginica
+2.9567515352594244E-6,0.07228280087561359,0.927714242372851,Iris-virginica,Iris-virginica
+2.3982340907419163E-7,0.042374076481183916,0.9576256836954069,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json
new file mode 100644
index 000000000..88cf740e6
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 1,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415601,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/1/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/1/metadata.json",
+    "result_token": "3538d1b4-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpzzp9l29e",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv
new file mode 100644
index 000000000..1837d94ea
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9977407400831685,0.00225925991682751,4.015165004779329E-15,Iris-setosa,Iris-setosa
+0.9998127799097366,1.872200902633975E-4,4.4069911448376E-17,Iris-setosa,Iris-setosa
+0.9991029079294788,8.970920705179842E-4,3.36945787249407E-15,Iris-setosa,Iris-setosa
+0.9988618905736355,0.0011381094263636813,9.755423863959563E-16,Iris-setosa,Iris-setosa
+0.9991718877459559,8.281122540317876E-4,1.2331412068001223E-14,Iris-setosa,Iris-setosa
+1.4444276601415674E-5,0.9586804544702732,0.04130510125312546,Iris-versicolor,Iris-versicolor
+0.0013080421910475607,0.9914876185496758,0.0072043392592766094,Iris-versicolor,Iris-versicolor
+9.165272335157342E-5,0.9926865448386248,0.007221802438023577,Iris-versicolor,Iris-versicolor
+4.000656682613745E-5,0.9403412635245068,0.05961872990866716,Iris-versicolor,Iris-versicolor
+4.556745505745815E-4,0.9862291776381161,0.013315147811309238,Iris-versicolor,Iris-versicolor
+2.823761394672141E-5,0.1259734017180567,0.8739983606679965,Iris-virginica,Iris-virginica
+6.633076885499847E-9,3.6946619151488527E-4,0.9996305271754083,Iris-virginica,Iris-virginica
+1.124808863216433E-8,0.005488528420360704,0.9945114603315506,Iris-virginica,Iris-virginica
+4.547823734480297E-8,0.0026021657037141105,0.9973977888180486,Iris-virginica,Iris-virginica
+3.4159926848071144E-8,0.012462663957173407,0.9875373018828998,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json
new file mode 100644
index 000000000..1932fecbe
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 2,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415602,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/2/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/2/metadata.json",
+    "result_token": "5bbdfe5e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmppxvl4v2c",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv
new file mode 100644
index 000000000..f700a8d34
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.999643185421844,3.5681457815609174E-4,2.230159041262659E-17,Iris-setosa,Iris-setosa
+0.7411777171360305,0.2588222828628354,1.1341170862473228E-12,Iris-setosa,Iris-setosa
+0.9894857732646907,0.010514226735307778,1.4471496430130413E-15,Iris-setosa,Iris-setosa
+0.9995220482574042,4.7795174259571756E-4,3.52685599190754E-18,Iris-setosa,Iris-setosa
+0.9988481135869727,0.001151886413027263,3.987099704316224E-17,Iris-setosa,Iris-setosa
+0.0013884526297230128,0.9940428944867407,0.004568652883536275,Iris-versicolor,Iris-versicolor
+0.005953079691957121,0.9940145236999208,3.239660812213889E-5,Iris-versicolor,Iris-versicolor
+0.010116816444659752,0.9893970388624135,4.8614469292683455E-4,Iris-versicolor,Iris-versicolor
+7.23459628127077E-4,0.99243180940297,0.006844730968902987,Iris-versicolor,Iris-versicolor
+3.3659932451520145E-6,0.5553095425683089,0.4446870914384459,Iris-versicolor,Iris-versicolor
+1.2929154379113906E-12,7.240147885676274E-5,0.9999275985198504,Iris-virginica,Iris-virginica
+5.618995240209567E-8,0.002834747714656968,0.9971651960953906,Iris-virginica,Iris-virginica
+1.0168116273950983E-8,0.004773137506941937,0.9952268523249418,Iris-virginica,Iris-virginica
+1.1743944498400225E-12,3.7400521135450664E-5,0.9999625994776901,Iris-virginica,Iris-virginica
+3.555096876025579E-10,1.2269388421367656E-4,0.9998773057602767,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json
new file mode 100644
index 000000000..3ab354b47
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 3,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415603,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/3/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/3/metadata.json",
+    "result_token": "842080a6-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpju0mvnpz",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv
new file mode 100644
index 000000000..7888779c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9957981631401925,0.004201836859803345,4.248771336417623E-15,Iris-setosa,Iris-setosa
+0.9999242893981092,7.57106018908299E-5,7.555452439324001E-18,Iris-setosa,Iris-setosa
+0.9928309264483719,0.0071690735515837324,4.435073927352755E-14,Iris-setosa,Iris-setosa
+0.9981107676724997,0.0018892323274941557,6.2433444463775446E-15,Iris-setosa,Iris-setosa
+0.9992575208258436,7.424791741562481E-4,3.2743342900204155E-16,Iris-setosa,Iris-setosa
+0.00617078018723429,0.9937995194498253,2.9700362940417253E-5,Iris-versicolor,Iris-versicolor
+0.001856136695353521,0.9827598819837347,0.015383981320911793,Iris-versicolor,Iris-versicolor
+0.046721989468264224,0.9532569491771904,2.106135454538263E-5,Iris-versicolor,Iris-versicolor
+0.0023199782194149123,0.9971359443070136,5.440774735713279E-4,Iris-versicolor,Iris-versicolor
+1.2065477934082464E-5,0.4747201530513801,0.5252677814706859,Iris-virginica,Iris-versicolor
+1.1586735176370364E-9,4.4327508426450854E-4,0.9995567237570618,Iris-virginica,Iris-virginica
+8.213607614325511E-13,7.434115656964782E-5,0.9999256588426091,Iris-virginica,Iris-virginica
+6.260997775752786E-9,0.0015969357139136785,0.9984030580250886,Iris-virginica,Iris-virginica
+2.302723428633504E-10,1.6320985908881373E-5,0.9999836787838188,Iris-virginica,Iris-virginica
+5.077776188678325E-10,0.0022517395458113206,0.9977482599464109,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json
new file mode 100644
index 000000000..b31af78f4
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 4,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415604,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/4/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/4/metadata.json",
+    "result_token": "ae4c3d3e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpdd0mphh0",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv
new file mode 100644
index 000000000..ebb85511f
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9995671767145491,4.3282328545065474E-4,2.910634990710733E-16,Iris-setosa,Iris-setosa
+0.9982697586656699,0.0017302413343282233,1.835406762143692E-15,Iris-setosa,Iris-setosa
+0.9989863122091491,0.0010136877908451531,5.704916978743878E-15,Iris-setosa,Iris-setosa
+0.9996616547927483,3.3834520725136086E-4,3.9294020366326627E-16,Iris-setosa,Iris-setosa
+0.9998864533391787,1.1354666082124777E-4,6.7435833537977226E-18,Iris-setosa,Iris-setosa
+0.004219356285716925,0.9920461471456139,0.003734496568669032,Iris-versicolor,Iris-versicolor
+6.839164491040382E-5,0.9945115713106095,0.005420037044480197,Iris-versicolor,Iris-versicolor
+0.0017683621545000424,0.9872280359587889,0.011003601886711033,Iris-versicolor,Iris-versicolor
+1.7369587491859734E-4,0.996893550822284,0.0029327533027972675,Iris-versicolor,Iris-versicolor
+4.354506494472319E-4,0.9882308872752477,0.011333662075305109,Iris-versicolor,Iris-versicolor
+2.697533881630151E-10,1.8877887113828945E-4,0.9998112208591085,Iris-virginica,Iris-virginica
+1.1833692210982864E-5,0.3136286688694338,0.6863594974383552,Iris-virginica,Iris-virginica
+3.860662541464733E-6,0.10171920502722513,0.8982769343102335,Iris-virginica,Iris-virginica
+2.6305660152696093E-7,0.003017310004465517,0.996982426938933,Iris-virginica,Iris-virginica
+3.5401683179702347E-7,0.020439351531391092,0.9795602944517771,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json
new file mode 100644
index 000000000..e0acd031b
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 5,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415605,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/5/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/5/metadata.json",
+    "result_token": "d6bb539a-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp6cunar4k",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv
new file mode 100644
index 000000000..a7164f84c
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9981088020680684,0.0018911979257595212,6.1720155217412105E-12,Iris-setosa,Iris-setosa
+0.9975129251674513,0.002487074799287478,3.3261253297507724E-11,Iris-setosa,Iris-setosa
+0.9872452587941326,0.012754741121718594,8.414891781400091E-11,Iris-setosa,Iris-setosa
+0.9865050960521858,0.013494903905854394,4.1959818974613784E-11,Iris-setosa,Iris-setosa
+0.9936909152559753,0.006309084608078586,1.3594606117458185E-10,Iris-setosa,Iris-setosa
+1.885964249056789E-4,0.3237625504836179,0.6760488530914763,Iris-virginica,Iris-versicolor
+0.006735391686925818,0.9524508509807971,0.04081375733227718,Iris-versicolor,Iris-versicolor
+5.378093523395172E-5,0.5228690867359151,0.4770771323288508,Iris-versicolor,Iris-versicolor
+3.5547125401524113E-4,0.8535160413485426,0.1461284873974421,Iris-versicolor,Iris-versicolor
+0.001280556645189532,0.9617810748971887,0.036938368457621766,Iris-versicolor,Iris-versicolor
+7.107012029851008E-5,0.29850295172759583,0.7014259781521057,Iris-virginica,Iris-virginica
+3.3942170781234547E-7,0.00447238089710375,0.9955272796811885,Iris-virginica,Iris-virginica
+1.895791713128563E-7,0.006446713099335271,0.9935530973214934,Iris-virginica,Iris-virginica
+4.3570097930097914E-4,0.3231055513427694,0.6764587476779297,Iris-virginica,Iris-virginica
+9.267662568271636E-5,0.05987127606163897,0.9400360473126782,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json
new file mode 100644
index 000000000..37ac517da
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 6,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415606,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/6/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/6/metadata.json",
+    "result_token": "fe15ff3a-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp_hfr1i3w",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv
new file mode 100644
index 000000000..3c1517540
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9986825173040808,0.0013174826959192415,2.8050853758906494E-17,Iris-setosa,Iris-setosa
+0.9998816031149136,1.1839688508629242E-4,9.723844873423558E-20,Iris-setosa,Iris-setosa
+0.9999856048453758,1.4395154624238886E-5,3.7272699996634135E-22,Iris-setosa,Iris-setosa
+0.9958896321731253,0.004110367826874745,9.1106915338899E-18,Iris-setosa,Iris-setosa
+0.9994195014384868,5.804985615131422E-4,8.352468371125052E-19,Iris-setosa,Iris-setosa
+1.0187624255752032E-4,0.9808217812619279,0.019076342495514632,Iris-versicolor,Iris-versicolor
+0.005455799247359012,0.9941389680709293,4.052326817118014E-4,Iris-versicolor,Iris-versicolor
+0.006332155970539919,0.9936285090877568,3.933494170343988E-5,Iris-versicolor,Iris-versicolor
+2.164701150526862E-4,0.9920704142697138,0.007713115615233376,Iris-versicolor,Iris-versicolor
+0.0012505917761682515,0.9987282670228536,2.1141200978234624E-5,Iris-versicolor,Iris-versicolor
+2.2087333569148024E-7,0.03154911382942394,0.9684506652972403,Iris-virginica,Iris-virginica
+8.458349099345969E-8,0.007958875829817774,0.9920410395866912,Iris-virginica,Iris-virginica
+3.5546202750827647E-12,7.378039045762072E-4,0.9992621960918691,Iris-virginica,Iris-virginica
+3.148087616934334E-18,1.0418835083791423E-7,0.9999998958116492,Iris-virginica,Iris-virginica
+3.935392414196986E-11,0.0011083045592472595,0.9988916954013989,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json
new file mode 100644
index 000000000..d18694044
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 7,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415607,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/7/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/7/metadata.json",
+    "result_token": "2416e3f2-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpa865dt_n",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv
new file mode 100644
index 000000000..f04ea4aae
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9990348975998995,9.651024001005388E-4,2.305810635776546E-17,Iris-setosa,Iris-setosa
+0.9994251478468725,5.748521531275645E-4,2.741195791809452E-18,Iris-setosa,Iris-setosa
+0.9974698813122362,0.0025301186877638043,1.4929300527492385E-17,Iris-setosa,Iris-setosa
+0.9980293757904823,0.001970624209517628,5.742383231957409E-18,Iris-setosa,Iris-setosa
+0.9989317358150057,0.0010682641849941218,3.939261654207119E-17,Iris-setosa,Iris-setosa
+3.700937950335641E-5,0.9867216516835606,0.013241338936936058,Iris-versicolor,Iris-versicolor
+0.0013687044804676057,0.9985112580034591,1.2003751607327173E-4,Iris-versicolor,Iris-versicolor
+0.006995911111617461,0.9682817728451619,0.02472231604322051,Iris-versicolor,Iris-versicolor
+1.3802427438593855E-4,0.9992264464192203,6.355293063938353E-4,Iris-versicolor,Iris-versicolor
+0.04709941690670438,0.9528951693622032,5.413731092497622E-6,Iris-versicolor,Iris-versicolor
+4.688538283391994E-11,5.7853607133277914E-5,0.9999421463459814,Iris-virginica,Iris-virginica
+1.6509551800831845E-5,0.7396217633784552,0.260361727069744,Iris-versicolor,Iris-virginica
+6.724194037868728E-4,0.3117625931928038,0.6875649874034093,Iris-virginica,Iris-virginica
+5.408383401537574E-10,4.153769951126476E-4,0.9995846224640491,Iris-virginica,Iris-virginica
+5.395915212183842E-11,4.1105559071968754E-4,0.9995889443553212,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json
new file mode 100644
index 000000000..0e52712c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 8,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415608,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22330,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/8/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/8/metadata.json",
+    "result_token": "4be39e52-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpgdovn6fi",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv
new file mode 100644
index 000000000..621acadad
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9922545822525884,0.007745417747410964,7.085353966139096E-16,Iris-setosa,Iris-setosa
+0.9963262691619229,0.0036737308380769234,1.5463726893802133E-16,Iris-setosa,Iris-setosa
+0.9997012711968896,2.9872880311049033E-4,1.7177502211229142E-17,Iris-setosa,Iris-setosa
+0.9982734171658221,0.0017265828341778042,1.953013147945192E-16,Iris-setosa,Iris-setosa
+0.9987322082865054,0.0012677917134945812,8.816555725807466E-18,Iris-setosa,Iris-setosa
+9.23865315845644E-4,0.9990174169021824,5.871778197195178E-5,Iris-versicolor,Iris-versicolor
+1.250278151239352E-4,0.9642979508172816,0.03557702136759442,Iris-versicolor,Iris-versicolor
+0.0013413343296529907,0.9299740756890498,0.06868458998129721,Iris-versicolor,Iris-versicolor
+8.35292503326207E-5,0.28583411631968514,0.7140823544299824,Iris-virginica,Iris-versicolor
+0.004834483157012863,0.9883085749200913,0.006856941922895849,Iris-versicolor,Iris-versicolor
+2.9816510566649863E-10,4.55874025606045E-5,0.9999544122992742,Iris-virginica,Iris-virginica
+1.2716794291346635E-5,0.229055275940032,0.7709320072656767,Iris-virginica,Iris-virginica
+2.440835419324771E-8,0.22493379307168823,0.7750661825199575,Iris-virginica,Iris-virginica
+6.1155599921910175E-9,0.0015434973500508815,0.9984564965343892,Iris-virginica,Iris-virginica
+2.1813458093942627E-11,1.0099649947803468E-4,0.9998990034787085,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json
new file mode 100644
index 000000000..fd5c3ffd8
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 9,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415609,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22330,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/9/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/9/metadata.json",
+    "result_token": "7468049e-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpyme85l1y",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv
new file mode 100644
index 000000000..d71e4bc7e
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9994560256672776,5.439593975257684E-4,1.4935196648718606E-8,Iris-setosa,Iris-setosa
+0.9991506482838198,8.493452971975329E-4,6.4189825553693556E-9,Iris-setosa,Iris-setosa
+0.9967761624035782,0.0032238257275304743,1.1868891386506383E-8,Iris-setosa,Iris-setosa
+0.9985515492213874,0.001448441407003948,9.371608537457098E-9,Iris-setosa,Iris-setosa
+0.9988119885708648,0.0011880061718968458,5.257238382408336E-9,Iris-setosa,Iris-setosa
+0.01529292273184757,0.6326831426347472,0.35202393463340526,Iris-versicolor,Iris-versicolor
+0.0028013111855947344,0.5669742812230935,0.43022440759131186,Iris-versicolor,Iris-versicolor
+0.005414416565513969,0.9799766510832394,0.014608932351246532,Iris-versicolor,Iris-versicolor
+0.0035872791673992248,0.9478077493917355,0.048604971440865216,Iris-versicolor,Iris-versicolor
+0.0045389422457486435,0.9914975512751514,0.003963506479099829,Iris-versicolor,Iris-versicolor
+3.750897792875574E-5,0.17921483171391367,0.8207476593081576,Iris-virginica,Iris-virginica
+3.166596517299125E-5,0.31433974404012566,0.6856285899947012,Iris-virginica,Iris-virginica
+4.694421902512631E-7,0.011834435964019752,0.98816509459379,Iris-virginica,Iris-virginica
+9.957182955200623E-6,0.09057334922636441,0.9094166935906803,Iris-virginica,Iris-virginica
+4.998346379369319E-6,0.207413278285669,0.7925817233679515,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/test_file_loading.py b/tests/unit/amlb/uploads/test_file_loading.py
new file mode 100644
index 000000000..76b47a067
--- /dev/null
+++ b/tests/unit/amlb/uploads/test_file_loading.py
@@ -0,0 +1,103 @@
+import os
+from collections import OrderedDict
+import pathlib
+
+import openml
+import pandas as pd
+import pytest
+
+from amlb.uploads import _load_predictions, _load_fold, _get_flow, _load_task_data, \
+    _extract_and_format_hyperparameter_configuration, _upload_results
+
+here = pathlib.Path(__file__).parent.absolute()
+resources = here / 'resources'
+
+iris_constant = resources / "iris_constant"
+iris_h2o = resources / "iris_h2o"
+
+
+@pytest.fixture(scope="module")
+def with_oml_test_server():
+    openml.config.start_using_configuration_for_example()
+    yield
+    openml.config.stop_using_configuration_for_example()
+
+
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__load_fold_loads_prediction_file(with_oml_test_server):
+    iris_task = openml.tasks.get_task(1196)
+    predictions = _load_fold(iris_constant, 0, iris_task)
+    assert isinstance(predictions, pd.DataFrame)
+    assert predictions.shape == (15, 8)
+    assert set(predictions.columns) == {"repeat", "fold", "index", "iris-setosa",
+                                        "iris-versicolor", "iris-virginica",
+                                        "predictions", "truth"}
+
+
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__load_fold_attaches_correct_index_information(with_oml_test_server):
+    iris_task = openml.tasks.get_task(1196)
+    predictions = _load_fold(iris_constant, 0, iris_task)
+    assert list(predictions["index"]) == [43, 14, 37, 23, 10, 99, 87, 97, 62, 92, 119,
+                                          111, 144, 116, 125]
+    assert all(predictions["fold"] == 0)
+    predictions = _load_fold(iris_constant, 1, iris_task)
+    assert list(predictions["index"]) == [49, 15, 47, 0, 44, 76, 89, 58, 54, 53, 149,
+                                          114, 112, 115, 139]
+    assert all(predictions["fold"] == 1)
+
+
+@pytest.mark.use_disk
+def test__load_predictions_loads_all_files():
+    predictions = _load_predictions(iris_constant)
+    assert isinstance(predictions, pd.DataFrame)
+    assert predictions.shape == (150, 8)
+    assert set(predictions.columns) == {"repeat", "fold", "index", "iris-setosa",
+                                        "iris-versicolor", "iris-virginica",
+                                        "predictions", "truth"}
+    assert set(predictions["fold"]) == set(range(10))
+
+
+@pytest.mark.use_disk
+def test__extract_flow_hyperparameter_configuration():
+    metadata = _load_task_data(iris_constant)
+    flow = _get_flow(metadata, sync_with_server=False)
+    parameters = _extract_and_format_hyperparameter_configuration(metadata, flow)
+
+    expected_parameters = [
+        OrderedDict([("oml:name", "max_runtime_seconds"), ("oml:value", 3600),
+                     ("oml:component", flow.id)]),
+        OrderedDict([("oml:name", "max_mem_size_mb"), ("oml:value", 2465),
+                     ("oml:component", flow.id)]),
+        OrderedDict(
+            [("oml:name", "cores"), ("oml:value", 8), ("oml:component", flow.id)]),
+        OrderedDict([("oml:name", "seed"), ("oml:value", 1362575752),
+                     ("oml:component", flow.id)]),
+    ]
+    assert parameters == expected_parameters
+
+
+@pytest.mark.openml_upload
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__upload_results_constant(with_oml_test_server):
+    run = _upload_results(iris_constant)
+    assert isinstance(run.id, int)
+
+
+@pytest.mark.openml_upload
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__upload_results_h2o(with_oml_test_server):
+    run = _upload_results(iris_h2o)
+    assert isinstance(run.id, int)
+
+
+@pytest.mark.skip(
+    "I can't think of a meaningful test that doesn't just verify that openml-python works as expected."
+)
+def test__get_flow(with_oml_test_server):
+    metadata = _load_task_data(iris_constant)
+    flow = _get_flow(metadata)