Repository: openml/automlbenchmark (language: python)

Issue:
Not enough precision in predict_duration in results.csv
We are [rounding](https://github.com/openml/automlbenchmark/blob/aee089164e8236e581672732c8bdc5ee6393d0ac/amlb/results.py#L172) to the first decimal for `predict_duration` so a lot of the results are getting rounded to 0.

Here, `"{:.1f}".format`, we need .3g instead of .1f.
fix for mlr3automl
makes PR #424 obsolete
https://download.mlplan.org/ does not have mlplan.zip
The discussion started in #463 but MlPlan is still failing.

Should it be this [file](https://download.mlplan.org/ailibs-0.9.0.tar.gz) (see https://download.mlplan.org/)?

Github actions is failing at this `mlplanweka` task.
EC2 AMIs: Outdated
The current EC2 AMis used in AMLB are very old (I believe they were created in ~2019):

```
  ec2:
    regions:                             #
      us-east-1:
        ami: ami-0ac019f4fcb7cb7e6
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      us-west-1:
        ami: ami-063aa838bd7631e0b
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      eu-west-1:
        ami: ami-00035f41c82244dab
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
      eu-central-1:
        ami: ami-0bdf93799014acdc4
        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
```

This leads to them having security vulnerabilities and thus are not able to be SSH'ed into when using an AWS account with proper security controls.

I would strongly recommend these AMI's be updated to 2023 AMIs.

I have tested the following us-east-1 AMI which runs successfully on AutoGluon:

```
    # Note: Added a new updated region image, old images no longer responded to SSH and didn't work correctly
    regions:                             #
      us-east-1:
        ami: ami-09cd747c78a9add63
        description: Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2023-02-07
```

I found this updated AMI by:

1. Going to [https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstances:](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstances:)
2. Clicking 'Browse more AMIs'
3. Under 'Quickstart AMIs', selecting 'Ubuntu Server 20.04 LTS (HVM), SSD Volume Type'
4. In the AMI from catalog view, you can see the AMI ID:
```
Amazon Machine Image (AMI)

ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230207
ami-09cd747c78a9add63
```
-> 
```
ami-09cd747c78a9add63
```

We can see that this is a recently created AMI:

```
Published

2023-02-07T22:43:00.000Z
```

We should be able to repeat this process for every region we want to support, and then update the `config.yaml` with the new AMIs.

Note: I updated to 20.04 from 18.04, but there is also an updated 18.04 AMI image too in case using Ubuntu 18.04 is preferred over 20.04.

```
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20230131
ami-0263e4deb427da90e
```
```
Published

2023-01-31T16:36:13.000Z
```


Discussion / hints:

Thanks! I can confirm it builds. It passes all validation tests except `census-income`, where it gives the following error:
```
[DEBUG] [amlb.utils.process:20:56:19.849] Running cmd `Rscript --vanilla -e ".libPaths('/bench/frameworks/mlr3automl/lib'); source('/bench/frameworks/mlr3automl/exec.R'); run('/input/org/openml/www/datasets/4535
/dataset_train_1.arff', '/input/org/openml/www/datasets/4535/dataset_test_1.arff', target.index = 42, 'classification', '/output/predictions/census-income/1/predictions.csv', 4, time.budget = 600, meta_results_f
ile='/output/meta_results.csv', seed='710346650', name='census-income')"` with input: None
[INFO] [amlb.print:20:56:20.497]
[INFO] [amlb.print:20:56:20.498] Attaching package: ‘mlr3extralearners’
[INFO] [amlb.print:20:56:20.498]
[INFO] [amlb.print:20:56:20.498] The following objects are masked from ‘package:mlr3’:
[INFO] [amlb.print:20:56:20.502]
[INFO] [amlb.print:20:56:20.503]     lrn, lrns
[INFO] [amlb.print:20:56:20.503]
[INFO] [amlb.print:20:56:20.503] Loading required package: paradox
[INFO] [amlb.print:20:56:31.487] [1] "Finished loading data after  9.65436553955078  seconds"
[INFO] [amlb.print:20:56:31.488] [1] "remaining budget:  590  seconds"
[INFO] [amlb.print:20:56:32.175] Error: Target column 'V42' must have at least two levels
[INFO] [amlb.print:20:56:32.176] Timing stopped at: 0.485 0.007 0.37
[INFO] [amlb.print:20:56:32.431]
[INFO] [amlb.print:20:56:32.431] Execution halted
[INFO] [amlb.print:20:56:32.431]
[INFO] [amlb.print:20:56:32.431]
[INFO] [amlb.print:20:56:32.432]
[DEBUG] [amlb.utils.process:20:56:32.432] [1] "Finished loading data after  9.65436553955078  seconds"
```

My best guess is that this is some data loading issue, could you still have a closer look?
thx for checking @PGijsbers, it's probably a glmnet specific error which occurs in some ege case situations. @a-hanf will have a look after his christmas holidays.
I had a look today and it seems unrelated to `mlr3automl` or `glmnet`.
I executed the benchmark locally and had a look at the data. After reading the training data here:
https://github.com/openml/automlbenchmark/blob/beef02403389809d6722dbc81dd4612910f24861/frameworks/mlr3automl/exec.R#L14
the training data set has only missing values in the labels column. I could imagine two reasons:
- AutoML benchmark saving the data as .arff is messing up
- the .arff parser is reading it incorrectly


For the first point - do other non-R frameworks read this from the .arff correctly? That would mean the second point is the critical one.
For the second point - do other R frameworks work correctly here? We should be able to load the data the same way as other frameworks then.

I don't know enough about .arff parsing / how the benchmark handles these data sets internally to debug this. @Coorsaa can you investigate further?
Seems like it works with the latest changes to `stable-v2`! 🎉 
@fmohr  is alan correct here? Has the file changed? Which MLPlan version is inside? More generally, do the AIlib versions reflect MLPlan version, or can different AIlib versions contain the same MLPlan version? From a usability/reproducibility standpoint we need to know what we are installing and from where. It would also be great if the old files would remain downloadable even when new versions are released.
Hi @alanwilter and @PGijsbers  . sorry for the delayed reply. In fact, there was an issue on our server due to an update of apache. The `mlplan.zip` is now there, and the build should work. Could you please confirm this?

@PGijsbers : `MLPlan` is not part of `AILibs` but just uses it as a dependency. Or did I miss your question?

The old files remain on our server. `mlplan.zip` always points to the newest version, which is currently `mlplan-0.2.3.zip`.
Thanks @fmohr, the download now works after I added this change in `frameworks/MLPlan/setup.sh`:

```bash
- wget -q https://download.mlplan.org/version/$VERSION -O $DOWNLOAD_DIR/$MLPLAN_ARC
+ wget -q --no-check-certificate https://download.mlplan.org/version/$VERSION -O 
```

Now @PGijsbers, it's the java jar that's failing with (see full log [here](https://github.com/openml/automlbenchmark/actions/runs/3130808647/jobs/5081490717#step:7:5361)):

```log
...
ERROR [main] [ai.libs.mlplan.cli.MLPlanCLI.generateCommandLine (ai.libs.mlplan.cli.MLPlanCLI:174)] - ERROR: Unable to parse command-line arguments [-f, /home/runner/.cache/openml/org/openml/www/datasets/61/dataset_train_0.arff, -p, /home/runner/.cache/openml/org/openml/www/datasets/61/dataset_test_0.arff, -t, 60, -ncpus, 2, -l, LOGLOSS, -m, weka, -s, 985826994, -ooab, /home/runner/work/automlbenchmark/automlbenchmark/results/mlplanweka.test.test.local.20220926T202513/mlplan_out/iris/0/predictions.csv, -os, /home/runner/work/automlbenchmark/automlbenchmark/results/mlplanweka.test.test.local.20220926T202513/mlplan_out/iris/0/statistics.json, -tmp, /tmp/tmpzrfbcjbk] due to exception.
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -os
...
```

Hi, looking at the detailed logs, I believe that's still something on our side. Apparently an unsupported option in the main routine of the CLI. @mwever could you confirm on this one pls?

I would imagine that this has already been solved in a newer version of the MLPlan CLI, and we just need to update this on the download page.
The file downloaded is `mlplan-0.2.3.zip` and its content is `mlplan-cli-0.0.1.jar`. Is that expected?
Yes that's expected. 0.2.3 is the version of MLPlan itself, 0.0.1 is the version of the CLI (a bit confusing, admittedly).
```bash
$ java -jar mlplan-cli-0.0.1.jar -h                                                                                                                                                                                       Called ML-Plan CLI with the following params: >[-h]<
usage: java -jar <mlplan.jar>
ML-Plan CLI v0.0.1-alpha
================================
 -f,--datasetFit <arg>                             The dataset file in ARFF format used for searching an appropriate pipeline.
 -h,--help                                         Provides an overview of available parameters and their description.
 -l,--evaluationMeasure <arg>                      The loss function to be used for internally assessing a candidate's performance.
                                                   Note that loss functions are problem specific, i.e. for single-label classification, regression or multi-label classification allow for different loss functions respectively. Score functions are automatically transformed into a loss function.
                                                   Depending on the chosen module, the following options are available to be configured.
                                                   weka: AUC, F1, PRECISION, RECALL, ERRORRATE, LOGLOSS
                                                   sklearn-rul: ASYMMETRIC_LOSS, MEAN_ABSOLUTE_ERROR, MEAN_ABSOLUTE_PERCENTAGE_ERROR, MEAN_SQUARED_ERROR, ROOT_MEAN_SQUARED_ERROR, R2, ROOT_MEAN_SQUARED_LOGARITHM_ERROR
                                                   weka-regression: ASYMMETRIC_LOSS, MEAN_ABSOLUTE_ERROR, MEAN_ABSOLUTE_PERCENTAGE_ERROR, MEAN_SQUARED_ERROR, ROOT_MEAN_SQUARED_ERROR, R2, ROOT_MEAN_SQUARED_LOGARITHM_ERROR
                                                   sklearn-regression: ASYMMETRIC_LOSS, MEAN_ABSOLUTE_ERROR, MEAN_ABSOLUTE_PERCENTAGE_ERROR, MEAN_SQUARED_ERROR, ROOT_MEAN_SQUARED_ERROR, R2, ROOT_MEAN_SQUARED_LOGARITHM_ERROR
                                                   sklearn: AUC, F1, PRECISION, RECALL, ERRORRATE, LOGLOSS
                                                   sklearn-unlimited: AUC, F1, PRECISION, RECALL, ERRORRATE, LOGLOSS
                                                   weka-tiny: AUC, F1, PRECISION, RECALL, ERRORRATE, LOGLOSS
 -m,--module <arg>                                 The ML-Plan module to be used.(Default: weka)
                                                   weka, sklearn-rul, weka-regression, sklearn-regression, sklearn, sklearn-unlimited, weka-tiny
 -ncpus,--numCPUs <arg>                            The number of CPU cores to be used by ML-Plan.(Default: 4)
 -ooab,--outputOpenmlAutomlBenchmarkResult <arg>   Output the result of the AutoML process according to the OpenML AutoML Benchmark suite. Enabling this option requires a dataset for predicting to be provided.
 -openMLTask,--openMLTask <arg>                    The OpenML task and the fold to run ML-Plan on.
 -p,--datasetPredict <arg>                         The dataset file in arff format used for applying the found pipeline to.
 -pci,--positiveClassIndex <arg>                   The index of the class to be considered as positive (for asymmetric evaluation measures).(Default: 0)
 -pcn,--positiveClassName <arg>                    Path to a custom search space configuration file.
 -s,--seed <arg>                                   The randomness seed used for the pseudo random number generator.(Default: 42)
 -ssc,--searchSpaceConfiguraitonFile <arg>         Path to a custom search space configuration file.
 -t,--timeout <arg>                                The timeout for the entire ML-Plan run in seconds.(Default: 3600)
 -tc,--candidateTimeout <arg>                      The timeout for evaluating a single candidate in seconds.(Default: 300)
 -tn,--nodeEvaluationTimeout <arg>                 The timeout for evaluating a node in the search tree (in seconds), i.e., the timeout for all the random completions drawn below the current node. This timeout is usually set to the candidate timeout times the number of random completions. *Note*: The default is automatically adapted if this option is not set but the candidate evaluation timeout is configured.(Default: 900)
 -v,--visualize                                    Enable a visualization of the search tree of ML-Plan together with a live stream of observations.
===============================
Visit us at: https://mlplan.org
```

AMLB yet asks for `-os` and `-tmp` parameters, which I don't see above. Clearly `frameworks/MLPlan/exec.py` needs this `-os` (`statistics_file = os.path.join(mlplan_output_dir, 'statistics.json')`).

> @PGijsbers : MLPlan is not part of AILibs but just uses it as a dependency. Or did I miss your question?

I believe at the time I checked the only available download was AILibs which made me think that perhaps ML-Plan was absorbed into that package. It's clear now that they remain separate and it was just an issue with available files 👍  thanks for sorting this out. (I'll stay on the sideline for this issue as long as it looks like the issue is with ML-Plan itself)
@alanwilter, @PGijsbers , we now have the updated version in. At least on our end it also worked. Could you please confirm this? 
I did work for AMLB git actions, see full log here: https://github.com/openml/automlbenchmark/actions/runs/3145334048/jobs/5112508133

despite some error messages.

Mine dataset showed similar errors but failed to give a result. I will investigate my case but I think the java jar is working now.
Rerun my case and now it worked. I think this ticket can be close now. Many thanks!
Great, thanks @alanwilter  for the constructive feedback!
Thanks @alanwilter and @fmohr!
Is there a good reason to go with 20.04 over 22.04? The latter has two additional years of support which sounds great. So we would instead use `ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230208`(`ami-0557a15b87f6559cf)` or equivalent.
I think 22.04 should be fine, I don't personally know the difference between the versions of Ubuntu.

Fix patch:
diff --git a/.dockerignore b/.dockerignore
index 7602493a2..31ba7fb3b 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -1,5 +1,13 @@
 .git
 
+# files not required for running the benchmark
+.pytest_cache
+amlb_report
+docs
+examples
+reports
+tests
+
 # generated files/folders
 .ipynb_checkpoints
 logs
diff --git a/.github/scripts/version-set.sh b/.github/scripts/version-set.sh
index 734f64144..3f67a039e 100755
--- a/.github/scripts/version-set.sh
+++ b/.github/scripts/version-set.sh
@@ -3,7 +3,7 @@ VERSION="$1"
 PUSH=${2:-"no-push"}
 ADD_COMMIT=${3:-"add-commit"}
 
-VERSION_FILE='app/__version__.py'
+VERSION_FILE='amlb/__version__.py'
 VERSION_PLACEHOLDER='__version__ =\s+"(.*)"$'
 DEV_PLACEHOLDER='_dev_version =\s+"(.*)"$'
 
diff --git a/.github/workflows/run_all_frameworks.yml b/.github/workflows/run_all_frameworks.yml
index 542df1d48..b11ce4629 100644
--- a/.github/workflows/run_all_frameworks.yml
+++ b/.github/workflows/run_all_frameworks.yml
@@ -15,7 +15,7 @@ jobs:
       skip_baseline: ${{ steps.find-required-tests.outputs.skip_baseline }}
       skip_evaluation: ${{ steps.find-required-tests.outputs.skip_evaluation }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: pull base branch
         run: |
           git fetch --unshallow origin $GITHUB_BASE_REF
@@ -52,7 +52,7 @@ jobs:
           skip_evaluation=0
           if [[ $is_common -eq 0 ]];
           then
-            FRAMEWORKS='["autogluon", "autosklearn", "autoxgboost", "gama", "h2oautoml", "mlplanweka", "mlr3automl", "tpot", "tunedrandomforest"]'
+            FRAMEWORKS='["autogluon", "autosklearn", "gama", "h2oautoml", "mlplanweka", "mlr3automl", "tpot", "tunedrandomforest"]'
             TASKS='["iris", "kc2", "cholesterol"]'
             BENCHMARK='["test"]'
           else
@@ -90,17 +90,17 @@ jobs:
         task: [iris, kc2, cholesterol]
       fail-fast:  false
     steps:
-      - uses: actions/checkout@v2
-      - name: Setup Python 3.8
-        uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
         with:
-          python-version: '3.8'
+          python-version: '3.9'
       - name: Create venv
         run: python -m venv venv
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         id: cache
         with:
-          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.8/site-packages
+          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
           key: pip-v3-${{ hashFiles('**/requirements.txt') }}
           restore-keys: |
             pip-v3-
@@ -126,24 +126,24 @@ jobs:
 
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.9]
         framework: ${{ fromJson(needs.detect_changes.outputs.frameworks) }}
         task: ${{ fromJson(needs.detect_changes.outputs.tasks) }}
         benchmark: ${{ fromJson(needs.detect_changes.outputs.benchmark) }}
       fail-fast:  true  # not sure about this one, but considering the big workload it might be nicer
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Setup Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
     - name: Create venv
       run: python -m venv venv
-    - uses: actions/cache@v2
+    - uses: actions/cache@v3
       id: cache
       with:
-        path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.8/site-packages
+        path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
         key: pip-v3-${{ hashFiles('**/requirements.txt') }}
         restore-keys: |
           pip-v3-
@@ -152,8 +152,10 @@ jobs:
       run: |
         source venv/bin/activate
         python -m pip install --upgrade pip
-        pip install -r requirements.txt
+        python -m pip install -r requirements.txt
     - name: Run ${{ matrix.framework }} on ${{ matrix.task }}
       run: |
         source venv/bin/activate
         python runbenchmark.py ${{ matrix.framework }} ${{ matrix.benchmark }} test -f 0 -t ${{ matrix.task }} -e
+      env:
+        GITHUB_PAT: ${{ secrets.PUBLIC_ACCESS_GITHUB_PAT }}
diff --git a/.github/workflows/runbenchmark.yml b/.github/workflows/runbenchmark.yml
new file mode 100644
index 000000000..20bde27af
--- /dev/null
+++ b/.github/workflows/runbenchmark.yml
@@ -0,0 +1,38 @@
+name: Run `runbenchmark.py`
+
+on:
+  workflow_dispatch:
+    inputs:
+      python_version:
+        default: '3.9'
+      os:
+        default: 'ubuntu-latest'
+      framework:
+        default: 'randomforest'
+      options:
+        default: ''
+
+jobs:
+  run_benchmark:
+    runs-on: ${{ inputs.os }}
+    steps:
+      - name: Runs on ${{ inputs.os }}
+        run: echo Preparing to benchmark ${{ inputs.framework }} ${{ inputs.options }}
+      - name: Checkout ${{ github.ref_name }}
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.ref_name }}
+      - name: Install Py${{ inputs.python_version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ inputs.python_version }}
+      - name: Install benchmark dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -r requirements.txt
+      - name: Install ${{ inputs.framework }}
+        run: python runbenchmark.py ${{ inputs.framework }} -s only
+        env:
+          GITHUB_PAT: ${{ secrets.PUBLIC_ACCESS_GITHUB_PAT }}
+      - name: Benchmark ${{ inputs.framework }}
+        run: python runbenchmark.py ${{ inputs.framework }} ${{ inputs.options }}
diff --git a/.github/workflows/versioning-reset.yml b/.github/workflows/versioning-reset.yml
index 63fbf6424..8900a377e 100644
--- a/.github/workflows/versioning-reset.yml
+++ b/.github/workflows/versioning-reset.yml
@@ -7,7 +7,7 @@ on:
       - main
       - master
     paths:
-      - 'app/__version__.py'
+      - 'amlb/__version__.py'
 
   workflow_dispatch:
 
diff --git a/.gitignore b/.gitignore
index e3cad6b3e..bc9c76adc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,6 +16,7 @@ venv/
 .idea/
 *.iml
 *.swp
+launch.json
 
 # tmp files
 .ipynb_checkpoints/
@@ -47,3 +48,4 @@ Singularityfile
 #whitelist
 !reports/**/*.csv
 !reports/**/*.png
+!docs/img/**/*.png
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index b4bc78a5d..000000000
--- a/.travis.yml
+++ /dev/null
@@ -1,12 +0,0 @@
-language: python
-python:
-  - "3.7.9"
-
-before_install:
-  - python -m pip install --upgrade pip
-install:
-  - pip install -r requirements.txt
-  - pip install -r requirements-dev.txt
-
-script:
-  - python -m pytest -m "not stress"
diff --git a/amlb/__version__.py b/amlb/__version__.py
index 46945d24b..8c165df2f 100644
--- a/amlb/__version__.py
+++ b/amlb/__version__.py
@@ -1,3 +1,3 @@
 # major.minor.patch[-label]
 _dev_version = "dev"
-__version__ = "2.0.6"
+__version__ = "dev"
diff --git a/amlb/benchmark.py b/amlb/benchmark.py
index 4fbc93a3f..572dc91a3 100644
--- a/amlb/benchmark.py
+++ b/amlb/benchmark.py
@@ -7,6 +7,7 @@
 - run the jobs.
 - collect and save results.
 """
+import time
 from copy import copy
 from enum import Enum
 from importlib import import_module, invalidate_caches
@@ -24,8 +25,9 @@
 from .datautils import read_csv
 from .resources import get as rget, config as rconfig, output_dirs as routput_dirs
 from .results import ErrorResult, Scoreboard, TaskResult
-from .utils import Namespace as ns, OSMonitoring, as_list, datetime_iso, flatten, json_dump, lazy_property, profile, repr_def, \
-    run_cmd, run_script, signal_handler, str2bool, str_sanitize, system_cores, system_memory_mb, system_volume_mb, touch
+from .utils import Namespace as ns, OSMonitoring, as_list, datetime_iso, file_lock, flatten, json_dump, \
+    lazy_property, profile, repr_def, run_cmd, run_script, signal_handler, str2bool, str_sanitize, \
+    system_cores, system_memory_mb, system_volume_mb, touch
 
 
 log = logging.getLogger(__name__)
@@ -209,22 +211,21 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
             results = self._run_jobs(jobs)
             log.info(f"Processing results for {self.sid}")
             log.debug(results)
-            if tasks is None:
-                scoreboard = self._process_results(results)
-            else:
-                for task_def in task_defs:
-                    task_results = filter(lambda res: res.result is not None and res.result.task == task_def.name, results)
-                    scoreboard = self._process_results(task_results, task_name=task_def.name)
-            return scoreboard
+
+            if not rconfig().results.incremental_save:
+                self._process_results(results)
+            return self._results_summary()
         finally:
             self.cleanup()
 
     def _create_job_runner(self, jobs):
+        on_new_result = self._process_results if rconfig().results.incremental_save else None
         if self.parallel_jobs == 1:
-            return SimpleJobRunner(jobs)
+            return SimpleJobRunner(jobs, on_new_result=on_new_result)
         else:
-            # return ThreadPoolExecutorJobRunner(jobs, self.parallel_jobs)
-            return MultiThreadingJobRunner(jobs, self.parallel_jobs,
+            return MultiThreadingJobRunner(jobs,
+                                           on_new_result=on_new_result,
+                                           parallel_jobs=self.parallel_jobs,
                                            delay_secs=rconfig().job_scheduler.delay_between_jobs,
                                            done_async=True)
 
@@ -254,10 +255,6 @@ def on_interrupt(*_):
             pass
         finally:
             results = self.job_runner.results
-
-        for res in results:
-            if res.result is not None and math.isnan(res.result.duration):
-                res.result.duration = res.duration
         return results
 
     def _benchmark_tasks(self):
@@ -327,34 +324,45 @@ def _skip_job(self, task_def, fold):
 
         return False
 
-    def _process_results(self, results, task_name=None):
+    def _process_results(self, results):
+        if not isinstance(results, list):
+            results = [results]
         scores = list(filter(None, flatten([res.result for res in results])))
         if len(scores) == 0:
             return None
 
-        board = (Scoreboard(scores,
-                            framework_name=self.framework_name,
-                            task_name=task_name,
-                            scores_dir=self.output_dirs.scores) if task_name
-                 else Scoreboard(scores,
-                                 framework_name=self.framework_name,
-                                 benchmark_name=self.benchmark_name,
-                                 scores_dir=self.output_dirs.scores))
+        for res in results:
+            if math.isnan(res.result.duration):
+                res.result.duration = res.duration
 
-        if rconfig().results.save:
-            self._save(board)
-
-        log.info("Summing up scores for current run:\n%s",
-                 board.as_printable_data_frame(verbosity=2).dropna(how='all', axis='columns').to_string(index=False))
-        return board.as_data_frame()
+        board = Scoreboard(scores, scores_dir=self.output_dirs.scores)
+        self._save(board)
+        return board
 
     def _save(self, board):
         board.save(append=True)
-        self._append(board)
-
-    def _append(self, board):
-        Scoreboard.all().append(board).save()
-        Scoreboard.all(rconfig().output_dir).append(board).save()
+        self._save_global(board)
+
+    def _save_global(self, board):
+        # Scoreboard.all().append(board).save()
+        if rconfig().results.global_save:
+            global_board = Scoreboard.all(rconfig().output_dir, autoload=False)
+            dest_path = global_board.path
+            timeout = rconfig().results.global_lock_timeout
+            try:
+                with file_lock(dest_path, timeout=timeout):
+                    global_board.load().append(board).save()
+            except TimeoutError:
+                log.exception("Failed to acquire the lock on `%s` after %ss: "
+                              "the partial board `%s` could not be appended to `%s`",
+                              dest_path, timeout, board.path, dest_path)
+
+    def _results_summary(self, scoreboard=None):
+        board = scoreboard or Scoreboard.all(self.output_dirs.scores)
+        results = board.as_printable_data_frame(verbosity=2)
+        log.info("Summing up scores for current run:\n%s",
+                 results.dropna(how='all', axis='columns').to_string(index=False))
+        return board.as_data_frame()
 
     @lazy_property
     def output_dirs(self):
@@ -371,14 +379,16 @@ def _is_task_enabled(task_def):
 
 class TaskConfig:
 
-    def __init__(self, name, fold, metrics, seed,
+    def __init__(self, name, openml_task_id, test_server, fold, metrics, seed,
                  max_runtime_seconds, cores, max_mem_size_mb, min_vol_size_mb,
-                 input_dir, output_dir):
+                 input_dir, output_dir, tag, command, git_info, measure_inference_time: bool = False):
         self.framework = None
         self.framework_params = None
         self.framework_version = None
         self.type = None
         self.name = name
+        self.openml_task_id = openml_task_id
+        self.test_server = test_server
         self.fold = fold
         self.metrics = [metrics] if isinstance(metrics, str) else metrics
         self.seed = seed
@@ -389,19 +399,29 @@ def __init__(self, name, fold, metrics, seed,
         self.input_dir = input_dir
         self.output_dir = output_dir
         self.output_predictions_file = os.path.join(output_dir, "predictions.csv")
+        self.tag = tag
+        self.command = command
+        self.git_info = git_info
+        self.measure_inference_time = measure_inference_time
         self.ext = ns()  # used if frameworks require extra config points
 
     def __setattr__(self, name, value):
         if name == 'metrics':
             self.metric = value[0] if isinstance(value, list) else value
         elif name == 'max_runtime_seconds':
-            self.job_timeout_seconds = min(value * 2,
-                                           value + rconfig().benchmarks.overhead_time_seconds)
+            inference_time_extension = 0
+            if rconfig().inference_time_measurements.enabled:
+                inference_time_extension = rconfig().inference_time_measurements.additional_job_time
+            self.job_timeout_seconds = min(value * 2 + inference_time_extension,
+                                           value + rconfig().benchmarks.overhead_time_seconds + inference_time_extension)
         super().__setattr__(name, value)
 
     def __json__(self):
         return self.__dict__
 
+    def __repr__(self):
+        return repr_def(self)
+
     def estimate_system_params(self):
         on_unfulfilled = rconfig().benchmarks.on_unfulfilled_constraint
         mode = re.split(r"\W+", rconfig().run_mode, maxsplit=1)[0]
@@ -457,6 +477,7 @@ def __init__(self, benchmark: Benchmark, task_def, fold):
         self.fold = fold
         self.task_config = TaskConfig(
             name=task_def.name,
+            openml_task_id=task_def.openml_task_id,
             fold=fold,
             metrics=task_def.metric,
             seed=rget().seed(fold),
@@ -466,6 +487,11 @@ def __init__(self, benchmark: Benchmark, task_def, fold):
             min_vol_size_mb=task_def.min_vol_size_mb,
             input_dir=rconfig().input_dir,
             output_dir=benchmark.output_dirs.session,
+            test_server=rget().config.test_server,
+            tag=rget().config.__dict__.get("tag"),
+            command=rget().config.command,
+            git_info=rget().git_info,
+            measure_inference_time=rconfig().inference_time_measurements.enabled,
         )
         # allowing to override some task parameters through command line, e.g.: -Xt.max_runtime_seconds=60
         if rconfig()['t'] is not None:
@@ -486,7 +512,9 @@ def load_data(self):
             # TODO
             raise NotImplementedError("OpenML datasets without task_id are not supported yet.")
         elif hasattr(self._task_def, 'dataset'):
-            self._dataset = Benchmark.data_loader.load(DataSourceType.file, dataset=self._task_def.dataset, fold=self.fold)
+            dataset_name_and_config = copy(self._task_def.dataset)
+            dataset_name_and_config.name = self._task_def.name
+            self._dataset = Benchmark.data_loader.load(DataSourceType.file, dataset=dataset_name_and_config, fold=self.fold)
         else:
             raise ValueError("Tasks should have one property among [openml_task_id, openml_dataset_id, dataset].")
 
@@ -519,7 +547,12 @@ def run(self):
                              predictions_dir=self.benchmark.output_dirs.predictions)
         framework_def = self.benchmark.framework_def
         task_config = copy(self.task_config)
-        task_config.type = 'regression' if self._dataset.type == DatasetType.regression else 'classification'
+        if self._dataset.type == DatasetType.regression:
+            task_config.type = 'regression'
+        elif self._dataset.type == DatasetType.timeseries:
+            task_config.type = 'timeseries'
+        else:
+            task_config.type = 'classification'
         task_config.type_ = self._dataset.type.name
         task_config.framework = self.benchmark.framework_name
         task_config.framework_params = framework_def.params
@@ -538,7 +571,7 @@ def run(self):
 
         result = meta_result = None
         try:
-            log.info("Running task %s on framework %s with config:\n%s", task_config.name, self.benchmark.framework_name, repr_def(task_config))
+            log.info("Running task %s on framework %s with config:\n%s", task_config.name, self.benchmark.framework_name, task_config)
             json_dump(task_config, task_config.output_metadata_file, style='pretty')
             meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
         except Exception as e:
@@ -549,4 +582,3 @@ def run(self):
         finally:
             self._dataset.release()
         return results.compute_score(result=result, meta_result=meta_result)
-
diff --git a/amlb/benchmarks/openml.py b/amlb/benchmarks/openml.py
index 33d70d8fb..78aa01242 100644
--- a/amlb/benchmarks/openml.py
+++ b/amlb/benchmarks/openml.py
@@ -16,7 +16,7 @@ def is_openml_benchmark(benchmark: str) -> bool:
         supported_types = ['s', 't']
 
         if oml_id.isdecimal():
-            return domain == "openml" and oml_type in supported_types
+            return domain in ("openml", "test.openml") and oml_type in supported_types
     return False
 
 
@@ -26,6 +26,10 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
     path = None  # benchmark file does not exist on disk
     name = benchmark  # name is later passed as cli input again for containers, it needs to remain parsable
 
+    if domain == "test.openml":
+        log.debug("Setting openml server to the test server.")
+        openml.config.server = "https://test.openml.org/api/v1/xml"
+
     if openml.config.retry_policy != "robot":
         log.debug(
             "Setting openml retry_policy from '%s' to 'robot'." % openml.config.retry_policy)
@@ -38,7 +42,8 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
         data = openml.datasets.get_dataset(t.dataset_id, download_data=False, download_qualities=False)
         tasks = [Namespace(name=str_sanitize(data.name),
                            description=data.description,
-                           openml_task_id=t.id)]
+                           openml_task_id=t.id,
+                           id="{}.org/t/{}".format(domain, t.id))]
     elif oml_type == 's':
         log.info("Loading openml suite %s.", oml_id)
         suite = openml.study.get_suite(oml_id)
@@ -50,7 +55,8 @@ def load_oml_benchmark(benchmark: str) -> Tuple[str, Optional[str], List[Namespa
         for tid, did in zip(suite.tasks, suite.data):
             tasks.append(Namespace(name=str_sanitize(datasets.loc[did]['name']),
                                    description=f"{openml.config.server.replace('/api/v1/xml', '')}/d/{did}",
-                                   openml_task_id=tid))
+                                   openml_task_id=tid,
+                                   id="{}.org/t/{}".format(domain, tid)))
     else:
         raise ValueError(f"The oml_type is {oml_type} but must be 's' or 't'")
     return name, path, tasks
diff --git a/amlb/data.py b/amlb/data.py
index b329df27b..acca17841 100644
--- a/amlb/data.py
+++ b/amlb/data.py
@@ -43,7 +43,7 @@ def __init__(self, index, name, data_type, values=None, has_missing_values=False
         self.index = index
         self.name = name
         self.data_type = data_type.lower() if data_type is not None else None
-        self.values = self.normalize(values).tolist() if values is not None else None
+        self.values = values
         self.has_missing_values = has_missing_values
         self.is_target = is_target
         # print(self)
@@ -80,8 +80,16 @@ def one_hot_encoder(self):
     def normalize(self, arr):
         return np.char.lower(np.char.strip(np.asarray(arr).astype(str)))
 
+    @property
+    def values(self):
+        return self._values
+
+    @values.setter
+    def values(self, values):
+        self._values = self.normalize(values).tolist() if values is not None else None
+
     def __repr__(self):
-        return repr_def(self)
+        return repr_def(self, 'all')
 
 
 class Datasplit(ABC):
@@ -134,8 +142,7 @@ def y(self) -> DF:
     @lazy_property
     @profile(logger=log)
     def data_enc(self) -> AM:
-        data = np.where(self.data.notna(), self.data.values, None)
-        encoded_cols = [f.label_encoder.transform(data[:, f.index]) for f in self.dataset.features]
+        encoded_cols = [f.label_encoder.transform(self.data.iloc[:, f.index]) for f in self.dataset.features]
         # optimize mem usage : frameworks use either raw data or encoded ones,
         # so we can clear the cached raw data once they've been encoded
         self.release(['data', 'X', 'y'])
@@ -165,6 +172,7 @@ class DatasetType(Enum):
     binary = 1
     multiclass = 2
     regression = 3
+    timeseries = 4
 
 
 class Dataset(ABC):
diff --git a/amlb/datasets/file.py b/amlb/datasets/file.py
index bcb941ab2..ee92d3b1a 100644
--- a/amlb/datasets/file.py
+++ b/amlb/datasets/file.py
@@ -13,10 +13,10 @@
 from ..data import Dataset, DatasetType, Datasplit, Feature
 from ..datautils import read_csv, to_data_frame
 from ..resources import config as rconfig
-from ..utils import Namespace as ns, as_list, lazy_property, list_all_files, memoize, path_from_split, profile, split_path
-
-from .fileutils import download_file, is_archive, is_valid_url, unarchive_file, url_exists
+from ..utils import Namespace as ns, as_list, lazy_property, list_all_files, memoize, path_from_split, profile, repr_def, split_path
 
+from .fileutils import is_archive, is_valid_url, unarchive_file, get_file_handler
+from copy import deepcopy
 
 log = logging.getLogger(__name__)
 
@@ -33,7 +33,7 @@ def __init__(self, cache_dir=None):
     def load(self, dataset, fold=0):
         dataset = dataset if isinstance(dataset, ns) else ns(path=dataset)
         log.debug("Loading dataset %s", dataset)
-        paths = self._extract_train_test_paths(dataset.path if 'path' in dataset else dataset, fold=fold)
+        paths = self._extract_train_test_paths(dataset.path if 'path' in dataset else dataset, fold=fold, name=dataset['name'] if 'name' in dataset else None)
         assert fold < len(paths['train']), f"No training dataset available for fold {fold} among dataset files {paths['train']}"
         # seed = rget().seed(fold)
         # if len(paths['test']) == 0:
@@ -51,21 +51,28 @@ def load(self, dataset, fold=0):
         if ext == '.arff':
             return ArffDataset(train_path, test_path, target=target, features=features, type=type_)
         elif ext == '.csv':
-            return CsvDataset(train_path, test_path, target=target, features=features, type=type_)
+            if type_ is not None and DatasetType[type_] == DatasetType.timeseries and dataset['timestamp_column'] is None:
+                log.warning("Warning: For timeseries task setting undefined timestamp column to `timestamp`.")
+                dataset = deepcopy(dataset)
+                dataset['timestamp_column'] = "timestamp"
+            csv_dataset = CsvDataset(train_path, test_path, target=target, features=features, type=type_, timestamp_column=dataset['timestamp_column'] if 'timestamp_column' in dataset else None)
+            if csv_dataset.type == DatasetType.timeseries:
+                csv_dataset = self.extend_dataset_with_timeseries_config(csv_dataset, dataset)
+            return csv_dataset
         else:
             raise ValueError(f"Unsupported file type: {ext}")
 
-    def _extract_train_test_paths(self, dataset, fold=None):
+    def _extract_train_test_paths(self, dataset, fold=None, name=None):
         if isinstance(dataset, (tuple, list)):
             assert len(dataset) % 2 == 0, "dataset list must contain an even number of paths: [train_0, test_0, train_1, test_1, ...]."
             return self._extract_train_test_paths(ns(train=[p for i, p in enumerate(dataset) if i % 2 == 0],
                                                      test=[p for i, p in enumerate(dataset) if i % 2 == 1]),
-                                                  fold=fold)
+                                                  fold=fold, name=name)
         elif isinstance(dataset, ns):
-            return dict(train=[self._extract_train_test_paths(p)['train'][0]
+            return dict(train=[self._extract_train_test_paths(p, name=name)['train'][0]
                                if i == fold else None
                                for i, p in enumerate(as_list(dataset.train))],
-                        test=[self._extract_train_test_paths(p)['train'][0]
+                        test=[self._extract_train_test_paths(p, name=name)['train'][0]
                               if i == fold else None
                               for i, p in enumerate(as_list(dataset.test))])
         else:
@@ -116,14 +123,55 @@ def _extract_train_test_paths(self, dataset, fold=None):
                 assert len(paths) > 0, f"No dataset file found in {dataset}: they should follow the naming xxxx_train.ext, xxxx_test.ext or xxxx_train_0.ext, xxxx_test_0.ext, xxxx_train_1.ext, ..."
                 return paths
         elif is_valid_url(dataset):
-            cached_file = os.path.join(self._cache_dir, os.path.basename(dataset))
+            if name is None:
+                cached_file = os.path.join(self._cache_dir, os.path.basename(dataset))
+            else:
+                cached_file = os.path.join(self._cache_dir, name, os.path.basename(dataset))
             if not os.path.exists(cached_file):  # don't download if previously done
-                assert url_exists(dataset), f"Invalid path/url: {dataset}"
-                download_file(dataset, cached_file)
+                handler = get_file_handler(dataset)
+                assert handler.exists(dataset), f"Invalid path/url: {dataset}"
+                handler.download(dataset, dest_path=cached_file)
             return self._extract_train_test_paths(cached_file)
         else:
             raise ValueError(f"Invalid dataset description: {dataset}")
 
+    def __repr__(self):
+        return repr_def(self)
+
+
+    def extend_dataset_with_timeseries_config(self, dataset, dataset_config):
+        dataset = deepcopy(dataset)
+        dataset_config = deepcopy(dataset_config)
+        if dataset_config['id_column'] is None:
+            log.warning("Warning: For timeseries task setting undefined `id_column` to `item_id`.")
+            dataset_config['id_column'] = "item_id"
+        if dataset_config['forecast_range_in_steps'] is None:
+            log.warning("Warning: For timeseries task setting undefined `forecast_range_in_steps` to `1`.")
+            dataset_config['forecast_range_in_steps'] = "1"
+
+        dataset.timestamp_column=dataset_config['timestamp_column']
+        dataset.id_column=dataset_config['id_column']
+        dataset.forecast_range_in_steps=int(dataset_config['forecast_range_in_steps'])
+
+        train_seqs_lengths = dataset.train.X.groupby(dataset.id_column).count()
+        test_seqs_lengths = dataset.test.X.groupby(dataset.id_column).count()
+        forecast_range_in_steps_mean_diff_train_test = int((test_seqs_lengths - train_seqs_lengths).mean())
+        forecast_range_in_steps_max_min_train_test = int(min(int(test_seqs_lengths.min()), int(train_seqs_lengths.min()))) - 1
+        if not dataset.forecast_range_in_steps == forecast_range_in_steps_mean_diff_train_test:
+            msg = f"Warning: Forecast range {dataset.forecast_range_in_steps}, does not equal mean difference between test and train sequence lengths {forecast_range_in_steps_mean_diff_train_test}."
+            log.warning(msg)
+        if not (test_seqs_lengths - train_seqs_lengths).var().item() == 0.:
+            msg = f"Error: Not all sequences of train and test set have same sequence length difference."
+            raise ValueError(msg)
+        if dataset.forecast_range_in_steps > forecast_range_in_steps_mean_diff_train_test:
+            msg = f"Error: Forecast range {dataset.forecast_range_in_steps} longer than difference between test and train sequence lengths {forecast_range_in_steps_mean_diff_train_test}."
+            raise ValueError(msg)
+        if dataset.forecast_range_in_steps > forecast_range_in_steps_max_min_train_test:
+            msg = f"Error: Forecast range {dataset.forecast_range_in_steps} longer than minimum sequence length + 1, {forecast_range_in_steps_max_min_train_test}."
+            raise ValueError(msg)
+        return dataset
+
+
 
 class FileDataset(Dataset):
 
@@ -165,6 +213,9 @@ def _get_metadata(self, prop):
         meta = self._train.load_metadata()
         return meta[prop]
 
+    def __repr__(self):
+        return repr_def(self, 'all')
+
 
 class FileDatasplit(Datasplit):
 
@@ -211,11 +262,17 @@ def _set_feature_as_target(self, target: Feature):
         ds_type = self.dataset._type
         if ds_type and DatasetType[ds_type] in [DatasetType.binary, DatasetType.multiclass]:
             if not target.is_categorical():
-                log.warning("Forcing target column %s as 'category' for classification problems: was originally detected as '%s'.",
+                log.warning("Forcing target column `%s` as 'category' for classification problems: was originally detected as '%s'.",
                             target.name, target.data_type)
-                # target.data_type = 'category'
+                self._convert_to_categorical(target)
         target.is_target = True
 
+    def _convert_to_categorical(self, feature: Feature):
+        feature.data_type = 'category'
+
+    def __repr__(self):
+        return repr_def(self, 'all')
+
 
 class ArffDataset(FileDataset):
 
@@ -289,35 +346,44 @@ def release(self, properties=None):
 class CsvDataset(FileDataset):
 
     def __init__(self, train_path, test_path,
-                 target=None, features=None, type=None):
+                 target=None, features=None, type=None, timestamp_column=None):
         # todo: handle auto-split (if test_path is None): requires loading the training set, split, save
-        super().__init__(CsvDatasplit(self, train_path), CsvDatasplit(self, test_path),
+        super().__init__(None, None,
                          target=target, features=features, type=type)
+        self._train = CsvDatasplit(self, train_path, timestamp_column=timestamp_column)
+        self._test = CsvDatasplit(self, test_path, timestamp_column=timestamp_column)
         self._dtypes = None
 
 
 class CsvDatasplit(FileDatasplit):
 
-    def __init__(self, dataset, path):
+    def __init__(self, dataset, path, timestamp_column=None):
         super().__init__(dataset, format='csv', path=path)
         self._ds = None
+        self.timestamp_column = timestamp_column
 
     def _ensure_loaded(self):
         if self._ds is None:
             if self.dataset._dtypes is None:
-                df = read_csv(self.path)
+                df = read_csv(self.path, timestamp_column=self.timestamp_column)
                 # df = df.convert_dtypes()
                 dt_conversions = {name: 'category'
                                   for name, dtype in zip(df.dtypes.index, df.dtypes.values)
-                                  if pat.is_string_dtype(dtype) or pat.is_object_dtype(dtype)}
+                                  if pat.is_string_dtype(dtype)
+                                  or pat.is_object_dtype(dtype)
+                                  or (name == self.dataset._target
+                                      and self.dataset._type is not None
+                                      and DatasetType[self.dataset._type] in [DatasetType.binary, DatasetType.multiclass])
+                                  }
                 # we could be a bit more clever in the future and convert 'string' to category iff len(distinct values) << nrows
                 if dt_conversions:
                     df = df.astype(dt_conversions, copy=False)
 
                 self._ds = df
                 self.dataset._dtypes = self._ds.dtypes
+
             else:
-                self._ds = read_csv(self.path, dtype=self.dataset._dtypes.to_dict())
+                self._ds = read_csv(self.path, dtype=self.dataset._dtypes.to_dict(), timestamp_column=self.timestamp_column)
 
     @profile(logger=log)
     def load_metadata(self):
@@ -328,7 +394,7 @@ def load_metadata(self):
                                       else 'number' if pat.is_numeric_dtype(dt)
                                       else 'category' if pat.is_categorical_dtype(dt)
                                       else 'string' if pat.is_string_dtype(dt)
-                                      # else 'datetime' if pat.is_datetime64_dtype(dt)
+                                      else 'datetime' if pat.is_datetime64_dtype(dt)
                                       else 'object')
         features = [Feature(i, col, to_feature_type(dtypes[i]))
                     for i, col in enumerate(self._ds.columns)]
@@ -336,8 +402,9 @@ def load_metadata(self):
         for f in features:
             col = self._ds.iloc[:, f.index]
             f.has_missing_values = col.hasnans
+            # f.dtype = self._ds.dtypes[f.name]
             if f.is_categorical():
-                f.values = sorted(self._ds.dtypes[f.name].categories.values)
+                f.values = self._unique_values(f.name)
 
         target = self._find_target_feature(features)
         self._set_feature_as_target(target)
@@ -358,6 +425,11 @@ def release(self, properties=None):
         super().release(properties)
         self._ds = None
 
+    def _unique_values(self, col_name: str):
+        dt = self._ds.dtypes[col_name]
+        return sorted(dt.categories.values if hasattr(dt, 'categories')
+                      else self._ds[col_name].unique())
+
 
 class FileConverter:
     format = None
@@ -383,22 +455,24 @@ class ArffConverter(FileConverter):
 
     def _write_file(self, df, path):
         name = split_path(path).basename
+        description = f"Arff dataset file generated by automlbenchmark from {name}."
+        attributes = [(c,
+                       ('INTEGER' if pat.is_integer_dtype(dt)
+                        else 'REAL' if pat.is_float_dtype(dt)
+                        else 'NUMERIC' if pat.is_numeric_dtype(dt)
+                        # numeric categories need to be str: https://github.com/renatopp/liac-arff/issues/126
+                        else [str(cat) for cat in sorted(dt.categories.values)] if pat.is_categorical_dtype(dt)
+                        else 'STRING'
+                        ))
+                      for c, dt in zip(df.columns, df.dtypes)]
+        arff_data = dict(
+            description=description,
+            relation=name,
+            attributes=attributes,
+            data=df.values
+        )
         with open(path, 'w') as file:
-            description = f"Arff dataset file generated by automlbenchmark from {name}."
-            attributes = [(c,
-                           ('INTEGER' if pat.is_integer_dtype(dt)
-                            else 'REAL' if pat.is_float_dtype(dt)
-                            else 'NUMERIC' if pat.is_numeric_dtype(dt)
-                            else sorted(dt.categories.values) if pat.is_categorical_dtype(dt)
-                            else 'STRING'
-                            ))
-                          for c, dt in zip(df.columns, df.dtypes)]
-            arff.dump(dict(
-                description=description,
-                relation=name,
-                attributes=attributes,
-                data=df.values
-            ), file)
+            arff.dump(arff_data, file)
 
 
 class CsvConverter(FileConverter):
diff --git a/amlb/datasets/fileutils.py b/amlb/datasets/fileutils.py
index bce8cbc4f..f1b9eece9 100644
--- a/amlb/datasets/fileutils.py
+++ b/amlb/datasets/fileutils.py
@@ -2,39 +2,85 @@
 import os
 import shutil
 import tarfile
+import boto3
+from botocore.errorfactory import ClientError
 from urllib.error import URLError
 from urllib.parse import urlparse
-from urllib.request import Request, urlopen, urlretrieve
+from urllib.request import Request, urlopen
 import zipfile
 
 from ..utils import touch
 
 log = logging.getLogger(__name__)
 
-SUPPORTED_SCHEMES = ("http", "https")
 
+class FileHandler:
+    def exists(self, url): pass
+    def download(self, url, dest_path): pass
+
+
+class HttpHandler(FileHandler):
+    def exists(self, url):
+        head_req = Request(url, method='HEAD')
+        try:
+            with urlopen(head_req) as test:
+                return test.status == 200
+        except URLError as e:
+            log.error(f"Cannot access url %s: %s", url, e)
+            return False
+    
+    def download(self, url, dest_path):
+        touch(dest_path)
+        with urlopen(url) as resp, open(dest_path, 'wb') as dest:
+            shutil.copyfileobj(resp, dest)
+
+
+class S3Handler(FileHandler):
+    def exists(self, url):
+        s3 = boto3.client('s3')
+        bucket, key = self._s3_path_to_bucket_prefix(url)
+        try:
+            s3.head_object(Bucket=bucket, Key=key)
+            return True
+        except ClientError as e:
+            log.error(f"Cannot access url %s: %s", url, e)
+            return False
+        
+    def download(self, url, dest_path):
+        touch(dest_path)
+        s3 = boto3.resource('s3')
+        bucket, key = self._s3_path_to_bucket_prefix(url)
+        try:
+            s3.Bucket(bucket).download_file(key, dest_path)
+        except ClientError as e:
+            if e.response['Error']['Code'] == "404":
+                log.error("The object does not exist.")
+            else:
+                raise
+        
+    def _s3_path_to_bucket_prefix(self, s3_path):
+        s3_path_cleaned = s3_path.split('://', 1)[1]
+        bucket, prefix = s3_path_cleaned.split('/', 1)
+        return bucket, prefix
 
-def is_valid_url(url):
-    return urlparse(url).scheme in SUPPORTED_SCHEMES
 
+scheme_handlers = dict(
+    http=HttpHandler(),
+    https=HttpHandler(),
+    s3=S3Handler(),
+    s3a=S3Handler(),
+    s3n=S3Handler(),
+)
 
-def url_exists(url):
-    if not is_valid_url(url):
-        return False
-    head_req = Request(url, method='HEAD')
-    try:
-        with urlopen(head_req) as test:
-            return test.status == 200
-    except URLError as e:
-        log.error(f"Cannot access url %s: %s", url, e)
-        return False
+SUPPORTED_SCHEMES = list(scheme_handlers.keys())
 
 
-def download_file(url, dest_path):
-    touch(dest_path)
-    # urlretrieve(url, filename=dest_path)
-    with urlopen(url) as resp, open(dest_path, 'wb') as dest:
-        shutil.copyfileobj(resp, dest)
+def get_file_handler(url):
+    return scheme_handlers[urlparse(url).scheme]
+
+
+def is_valid_url(url):
+    return urlparse(url).scheme in SUPPORTED_SCHEMES
 
 
 def is_archive(path):
@@ -50,6 +96,24 @@ def unarchive_file(path, dest_folder=None):
             zf.extractall(path=dest_folder)
     elif tarfile.is_tarfile(path):
         with tarfile.open(path) as tf:
-            tf.extractall(path=dest_folder)
+            def is_within_directory(directory, target):
+                
+                abs_directory = os.path.abspath(directory)
+                abs_target = os.path.abspath(target)
+            
+                prefix = os.path.commonprefix([abs_directory, abs_target])
+                
+                return prefix == abs_directory
+            
+            def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
+            
+                for member in tar.getmembers():
+                    member_path = os.path.join(path, member.name)
+                    if not is_within_directory(path, member_path):
+                        raise Exception("Attempted Path Traversal in Tar File")
+            
+                tar.extractall(path, members, numeric_owner=numeric_owner) 
+                
+            
+            safe_extract(tf, path=dest_folder)
     return dest
-
diff --git a/amlb/datasets/openml.py b/amlb/datasets/openml.py
index d037252d8..657f49f99 100644
--- a/amlb/datasets/openml.py
+++ b/amlb/datasets/openml.py
@@ -2,21 +2,24 @@
 **openml** module implements the abstractions defined in **data** module
 to expose `OpenML<https://www.openml.org>`_ datasets.
 """
+import pathlib
 from abc import abstractmethod
 import copy
 import functools
 import logging
 import os
 import re
-from typing import Generic, Tuple, TypeVar, Union
+from typing import Generic, Tuple, TypeVar
 
 import arff
+import pandas as pd
 import pandas.api.types as pat
 import openml as oml
 import xmltodict
 
 from ..data import AM, DF, Dataset, DatasetType, Datasplit, Feature
-from ..resources import config as rconfig
+from ..datautils import impute_array
+from ..resources import config as rconfig, get as rget
 from ..utils import as_list, lazy_property, path_from_split, profile, split_path, unsparsify
 
 
@@ -66,6 +69,15 @@ def __init__(self, oml_task: oml.OpenMLTask, oml_dataset: oml.OpenMLDataset, fol
         self.fold = fold
         self._train = None
         self._test = None
+        self._nrows = None
+
+
+    @property
+    def nrows(self) -> int:
+        if self._nrows is None:
+            self._nrows = len(self._load_full_data(fmt='dataframe'))
+        return self._nrows
+
 
     @lazy_property
     def type(self):
@@ -92,6 +104,80 @@ def test(self):
         self._ensure_split_created()
         return self._test
 
+    def inference_subsample_files(self, fmt: str, with_labels: bool = False, scikit_safe: bool = False) -> list[Tuple[int, str]]:
+        """Generates n subsamples of size k from the test dataset in `fmt` data format.
+
+        We measure the inference time of the models for various batch sizes
+        (number of rows). We generate config.inference_time_measurements.repeats
+        subsamples for each of the config.inference_time_measurements.batch_sizes.
+
+        These subsamples are stored to file in the `fmt` format (parquet, arff, or csv).
+        The function returns a list of tuples of (batch size, file path).
+
+        Iff `with_labels` is true, the target column will be included in the split file.
+        Iff `scikit_safe` is true, categorical values are encoded and missing values
+        are imputed.
+        """
+        seed = rget().seed(self.fold)
+        batch_sizes = [
+            batch_size for batch_size in rconfig().inference_time_measurements.batch_sizes
+            if not (batch_size > self.nrows and rconfig().inference_time_measurements.limit_by_dataset_size)
+        ]
+        return [
+            (n, str(self._inference_subsample(fmt=fmt, n=n, seed=seed + i, with_labels=with_labels, scikit_safe=scikit_safe)))
+            for n in batch_sizes
+            for i, _ in enumerate(range(rconfig().inference_time_measurements.repeats))
+        ]
+
+    @profile(logger=log)
+    def _inference_subsample(self, fmt: str, n: int, seed: int = 0, with_labels: bool = False, scikit_safe: bool = False) -> pathlib.Path:
+        """ Write subset of `n` samples from the test split to disk in `fmt` format
+
+        Iff `with_labels` is true, the target column will be included in the split file.
+        Iff `scikit_safe` is true, categorical values are encoded and missing values
+        are imputed.
+        """
+        # Just a hack for now, the splitters all work specifically with openml tasks.
+        # The important thing is that we split to disk and can load it later.
+
+        # We should consider taking a stratified sample if n is large enough,
+        # inference time might differ based on class
+        if scikit_safe:
+            if with_labels:
+                _, data = impute_array(self.train.data_enc, self.test.data_enc)
+            else:
+                _, data = impute_array(self.train.X_enc, self.test.X_enc)
+
+            columns = self._test.data.columns if with_labels else self._test.X.columns
+            data = pd.DataFrame(data, columns=columns)
+        else:
+            data = self._test.data if with_labels else self._test.X
+
+        subsample = data.sample(
+            n=n,
+            replace=True,
+            random_state=seed,
+        )
+
+        _, test_path = self._get_split_paths()
+        test_path = pathlib.Path(test_path)
+        subsample_path = test_path.parent / f"{test_path.stem}_{n}_{seed}.{fmt}"
+        if fmt == "csv":
+            subsample.to_csv(subsample_path, header=True, index=False)
+        elif fmt == "arff":
+            ArffSplitter(self)._save_split(
+                subsample,
+                subsample_path,
+                name=f"{self._oml_dataset.name}_inference_{self.fold}_{n}"
+            )
+        elif fmt == "parquet":
+            subsample.to_parquet(subsample_path)
+        else:
+            msg = f"{fmt=}, but must be one of 'csv', 'arff', or 'parquet'."
+            raise ValueError(msg)
+
+        return subsample_path
+
     @lazy_property
     @profile(logger=log)
     def features(self):
diff --git a/amlb/datautils.py b/amlb/datautils.py
index f6bd2ef90..7946fdc1c 100644
--- a/amlb/datautils.py
+++ b/amlb/datautils.py
@@ -26,7 +26,7 @@
 log = logging.getLogger(__name__)
 
 
-def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dtype=None):
+def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dtype=None, timestamp_column=None):
     """
     read csv file to DataFrame.
 
@@ -37,13 +37,19 @@ def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dty
     :param header: if the columns header should be read.
     :param as_data_frame: if the result should be returned as a data frame (default) or a numpy array.
     :param dtype: data type for columns.
+    :param timestamp_column: column name for timestamp, to ensure dates are correctly parsed by pandas.
     :return: a DataFrame
     """
+    if dtype is not None and timestamp_column is not None and timestamp_column in dtype:
+            dtype = dtype.copy() # to avoid outer context manipulation
+            del dtype[timestamp_column]
+
     df = pd.read_csv(path,
                      nrows=nrows,
                      header=0 if header else None,
                      index_col=0 if index else None,
-                     dtype=dtype)
+                     dtype=dtype,
+                     parse_dates=[timestamp_column] if timestamp_column is not None else None)
     return df if as_data_frame else df.values
 
 
@@ -171,6 +177,9 @@ def __init__(self, type='label', target=True, encoded_type=int,
         else:
             raise ValueError("Encoder `type` should be one of {}.".format(['label', 'one-hot']))
 
+    def __repr__(self):
+        return repr_def(self)
+
     @property
     def _ignore_missing(self):
         return self.for_target or self.missing_policy == 'ignore'
@@ -211,8 +220,8 @@ def transform(self, vec, **params):
         :param params:
         :return:
         """
-        if log.isEnabledFor(5):  # logging.TRACE
-            log.debug("Transforming %s using %s", vec, repr_def(self))
+        if log.isEnabledFor(logging.TRACE):
+            log.debug("Transforming %s using %s", vec, self)
 
         return_value = lambda v: v
         if isinstance(vec, str):
diff --git a/amlb/defaults.py b/amlb/defaults.py
new file mode 100644
index 000000000..6d0bf35c5
--- /dev/null
+++ b/amlb/defaults.py
@@ -0,0 +1,12 @@
+import pathlib
+
+from openml.config import cache_directory
+
+from amlb.utils import Namespace as ns
+
+default_dirs = ns(
+    input_dir=cache_directory,
+    output_dir=str(pathlib.Path(__file__).parent.parent / "results"),
+    user_dir="~/.config/automlbenchmark",
+    root_dir=str(pathlib.Path(__file__).parent.parent)
+)
diff --git a/amlb/frameworks/definitions.py b/amlb/frameworks/definitions.py
index c1f050a19..3bd3c59a4 100644
--- a/amlb/frameworks/definitions.py
+++ b/amlb/frameworks/definitions.py
@@ -22,7 +22,6 @@ def load_framework_definitions(frameworks_file: Union[str, List[str]], config: N
     frameworks = _load_and_merge_framework_definitions(frameworks_file, config)
     for tag, defs in frameworks:
         _sanitize_and_add_defaults(defs, config)
-    log.debug("Available framework definitions:\n%s", frameworks)
     return frameworks
 
 
@@ -102,6 +101,9 @@ def _add_default_setup_env(framework):
 
 
 def _add_default_setup_args(framework):
+    if "setup_args" in framework and isinstance(framework.setup_args, list):
+        return
+
     if "setup_args" in framework and isinstance(framework.setup_args, str):
         framework.setup_args = [framework.setup_args]
     else:
diff --git a/amlb/job.py b/amlb/job.py
index 01dd24f37..6701f033d 100644
--- a/amlb/job.py
+++ b/amlb/job.py
@@ -14,6 +14,8 @@
 import queue
 import signal
 import threading
+from functools import partial
+from typing import Callable, List, Optional
 
 from .utils import Namespace, Timer, ThreadSafeCounter, InterruptTimeout, is_main_thread, raise_in_thread, signal_handler
 
@@ -64,7 +66,10 @@ def is_state_transition_ok(cls, old_state: State, new_state: State):
 
     printer = pprint.PrettyPrinter(indent=2, compact=True)
 
-    def __init__(self, name="", timeout_secs=None, priority=None, raise_on_failure=False):
+    def __init__(self, name: str = "",
+                 timeout_secs: int = -1,
+                 priority: Optional[int] = None,
+                 raise_on_failure: bool = False):
         """
 
         :param name:
@@ -195,18 +200,20 @@ class JobRunner:
         (State.stopping,    [State.stopped]),
         (State.stopped,     None)
     ]
+    END_Q = object()
 
     @classmethod
     def is_state_transition_ok(cls, old_state: State, new_state: State):
         allowed = next((head for tail, head in cls.state_machine if tail == old_state), None)
         return allowed and new_state in allowed
 
-    def __init__(self, jobs):
+    def __init__(self, jobs: List, on_new_result: Optional[Callable] = None):
         self.jobs = jobs
         self.results = []
         self.state = None
         self._queue = None
         self._last_priority = 0
+        self._on_new_result = on_new_result
         self.set_state(State.created)
 
     def start(self):
@@ -235,7 +242,7 @@ def stop_if_complete(self):
         if 0 < len(self.jobs) == len(self.results):
             self.stop()
 
-    def put(self, job, priority=None):
+    def put(self, job: Job, priority: Optional[int] = None):
         if self.state in [State.stopping, State.stopped]:
             return
         if priority is None:
@@ -248,7 +255,7 @@ def put(self, job, priority=None):
         else:
             log.warning("Ignoring job `%s`. Runner state: `%s`", job.name, self.state)
 
-    def reschedule(self, job, priority=None):
+    def reschedule(self, job: Job, priority: Optional[int] = None):
         if self.state not in [State.running]:
             return
         job.reschedule()
@@ -267,6 +274,11 @@ def set_state(self, state: State):
             log.exception("Error when handling state change to %s for job runner: %s", state, str(e))
         return not skip_default
 
+    def _add_result(self, result):
+        self.results.append(result)
+        if self._on_new_result is not None:
+            self._on_new_result(result)
+
     def __iter__(self):
         return self
 
@@ -289,7 +301,7 @@ def _run(self):
 
     def _stop(self):
         if self._queue:
-            self._queue.put((-1, None))
+            self._queue.put((-1, JobRunner.END_Q))
         jobs = self.jobs.copy()
         self.jobs.clear()
         for job in jobs:
@@ -301,19 +313,19 @@ def _on_state(self, state: State):
 
 class SimpleJobRunner(JobRunner):
 
-    def __init__(self, jobs):
-        super().__init__(jobs)
+    def __init__(self, jobs: List, on_new_result: Optional[Callable] = None):
+        super().__init__(jobs, on_new_result=on_new_result)
         self._interrupt = threading.Event()
 
     def _run(self):
         for job in self:
-            if job is None or self._interrupt.is_set():
+            if job is JobRunner.END_Q or self._interrupt.is_set():
                 break
             result = job.start()
             if job.state is State.rescheduling:
                 self.reschedule(job)
             else:
-                self.results.append(result)
+                self._add_result(result)
                 job.done()
             self.stop_if_complete()
 
@@ -328,16 +340,43 @@ class QueueingStrategy:
         keep_queue_full = 0
         enforce_job_priority = 1
 
-    def __init__(self, jobs, parallel_jobs=1, done_async=True, delay_secs=0,
+    def __init__(self, jobs: List,
+                 on_new_result: Optional[Callable] = None,
+                 parallel_jobs: int = 1,
+                 done_async: bool = True,
+                 delay_secs: int = 0,
                  queueing_strategy: QueueingStrategy = QueueingStrategy.keep_queue_full,
-                 use_daemons=False):
-        super().__init__(jobs)
+                 use_daemons: bool = False):
+        super().__init__(jobs, on_new_result=on_new_result)
         self.parallel_jobs = parallel_jobs
         self._done_async = done_async
         self._delay = delay_secs  # short sleep between enqueued jobs to make console more readable
         self._daemons = use_daemons
         self._queueing_strategy = queueing_strategy
         self._interrupt = threading.Event()
+        self._exec = None
+        self.futures = []
+
+    def _safe_call_from_exec(self, fn):
+        if self._exec:
+            future = self._exec.submit(fn)
+            self.futures.append(future)
+        else:
+            log.warning(
+                "Application is submitting a function while the thread executor is not running: executing the function in the calling thread.")
+            try:
+                fn()
+            except Exception as e:
+                log.exception(e)
+
+    def _add_result(self, result):
+        sup_call = partial(super()._add_result, result)
+        self._safe_call_from_exec(sup_call)
+
+    def stop_if_complete(self):
+        # Direct calls introduce a race condition by the queued '_add_result' calls
+        sup_call = super().stop_if_complete
+        self._safe_call_from_exec(sup_call)
 
     def _run(self):
         q = queue.Queue()
@@ -349,13 +388,13 @@ def worker():
                 job = q.get()
                 available_workers.dec()
                 try:
-                    if job is None or self._interrupt.is_set():
+                    if job is JobRunner.END_Q or self._interrupt.is_set():
                         break
                     result = job.start()
                     if job.state is State.rescheduling:
                         self.reschedule(job)
                     else:
-                        self.results.append(result)
+                        self._add_result(result)
                         if self._done_async:
                             job.done()
                     self.stop_if_complete()
@@ -387,7 +426,7 @@ def worker():
             q.maxsize = self.parallel_jobs  # resize to ensure that all workers can get a None job
             for _ in range(self.parallel_jobs):
                 try:
-                    q.put_nowait(None)     # stopping workers
+                    q.put_nowait(JobRunner.END_Q)     # stopping workers
                 except:
                     pass
             for thread in threads:
@@ -397,8 +436,17 @@ def worker():
                     job.done()
 
     def _on_state(self, state: State):
+        if state is State.starting:
+            self._exec = ThreadPoolExecutor(max_workers=1, thread_name_prefix="job_runner_exec_")
         if state is State.stopping:
             self._interrupt.set()
+            if self._exec is not None:
+                try:
+                    self._exec.shutdown(wait=True)
+                except:
+                    pass
+                finally:
+                    self._exec = None
 
 
 class MultiProcessingJobRunner(JobRunner):
diff --git a/amlb/resources.py b/amlb/resources.py
index d910587bc..808a7f954 100644
--- a/amlb/resources.py
+++ b/amlb/resources.py
@@ -132,11 +132,18 @@ def framework_definition(self, name, tag=None):
         if tag not in self._frameworks:
             raise ValueError("Incorrect tag `{}`: only those among {} are allowed.".format(tag, self.config.frameworks.tags))
         frameworks = self._frameworks[tag]
+        log.debug("Available framework definitions:\n%s", frameworks)
         framework = next((f for n, f in frameworks if n.lower() == lname), None)
+        # TODO: Clean up this workflow and error messaging as part of #518
+        base_framework = next((f for n, f in self._frameworks[default_tag] if n.lower() == lname), None)
+        if framework and framework['removed']:
+            raise ValueError(f"Framework definition `{name}` has been removed from the benchmark: {framework['removed']}")
+        if not framework and (base_framework and base_framework['removed']):
+            raise ValueError(f"Framework definition `{name}` has been removed from the benchmark: {base_framework['removed']}")
         if not framework:
-            raise ValueError("Incorrect framework `{}`: not listed in {}.".format(name, self.config.frameworks.definition_file))
+            raise ValueError(f"Incorrect framework `{name}`: not listed in {self.config.frameworks.definition_file}.")
         if framework['abstract']:
-            raise ValueError("Framework definition `{}` is abstract and cannot be run directly.".format(name))
+            raise ValueError(f"Framework definition `{name}` is abstract and cannot be run directly.")
         return framework, framework.name
 
     @lazy_property
@@ -217,7 +224,9 @@ def _validate_task(self, task, lenient=False):
                                  else None) or task.name) if task['dataset'] is not None
                           else None)
             if not lenient and task[conf] is None:
-                raise ValueError("task definition must contain one property among ['openml_task_id', 'dataset']")
+                raise ValueError("task definition must contain an ID or one property "
+                                 "among ['openml_task_id', 'dataset'] to create an ID, "
+                                 "but task definition is {task}".format(task=str(task)))
 
         conf = 'metric'
         if task[conf] is None:
@@ -273,6 +282,9 @@ def config():
 
 def output_dirs(root, session=None, subdirs=None, create=False):
     root = root if root is not None else '.'
+    if create and not os.path.exists(root):
+        touch(root, as_dir=True)
+
     dirs = Namespace(
         root=root,
         session=os.path.join(root, session) if session is not None else root
diff --git a/amlb/results.py b/amlb/results.py
index 2f547b4ec..6e1a5bc60 100644
--- a/amlb/results.py
+++ b/amlb/results.py
@@ -55,8 +55,8 @@ class Scoreboard:
     results_file = 'results.csv'
 
     @classmethod
-    def all(cls, scores_dir=None):
-        return cls(scores_dir=scores_dir)
+    def all(cls, scores_dir=None, autoload=True):
+        return cls(scores_dir=scores_dir, autoload=autoload)
 
     @classmethod
     def from_file(cls, path):
@@ -105,28 +105,34 @@ def load_df(file):
     def save_df(data_frame, path, append=False):
         exists = os.path.isfile(path)
         new_format = False
+        df = data_frame
         if exists:
-            df = read_csv(path, nrows=1)
-            new_format = list(df.columns) != list(data_frame.columns)
+            head = read_csv(path, nrows=1)
+            new_format = list(head.columns) != list(data_frame.columns)
         if new_format or (exists and not append):
             backup_file(path)
+        if new_format and append:
+            df = read_csv(path).append(data_frame)
         new_file = not exists or not append or new_format
         is_default_index = data_frame.index.name is None and not any(data_frame.index.names)
         log.debug("Saving scores to `%s`.", path)
-        write_csv(data_frame,
+        write_csv(df,
                   path=path,
                   header=new_file,
                   index=not is_default_index,
                   append=not new_file)
         log.info("Scores saved to `%s`.", path)
 
-    def __init__(self, scores=None, framework_name=None, benchmark_name=None, task_name=None, scores_dir=None):
+    def __init__(self, scores=None, framework_name=None, benchmark_name=None, task_name=None,
+                 scores_dir=None, autoload=True):
         self.framework_name = framework_name
         self.benchmark_name = benchmark_name
         self.task_name = task_name
         self.scores_dir = (scores_dir if scores_dir
                            else output_dirs(rconfig().output_dir, rconfig().sid, ['scores']).scores)
-        self.scores = scores if scores is not None else self._load()
+        self.scores = (scores if scores is not None
+                       else self.load_df(self.path) if autoload
+                       else None)
 
     @cached
     def as_data_frame(self):
@@ -155,24 +161,27 @@ def as_data_frame(self):
 
     @memoize
     def as_printable_data_frame(self, verbosity=3):
-        str_print = lambda val: '' if val in [None, '', 'None'] or (isinstance(val, float) and np.isnan(val)) else val
-        int_print = lambda val: int(val) if isinstance(val, float) and not np.isnan(val) else str_print(val)
-        num_print = lambda fn, val: None if isinstance(val, str) else fn(val)
+        str_print = lambda val: '' if val in [None, '', 'None'] or (isinstance(val, float) and np.isnan(val)) else str(val)
+        int_print = lambda val: int(val) if isinstance(val, (float, int)) and not np.isnan(val) else str_print(val)
 
         df = self.as_data_frame()
+        if df.empty:
+            return df
+
         force_str_cols = ['id']
         nanable_int_cols = ['fold', 'models_count', 'seed']
         low_precision_float_cols = ['duration', 'training_duration', 'predict_duration']
-        high_precision_float_cols = [col for col in df.select_dtypes(include=[np.float]).columns if col not in ([] + nanable_int_cols + low_precision_float_cols)]
+        high_precision_float_cols = [col for col in df.select_dtypes(include=[float]).columns if col not in ([] + nanable_int_cols + low_precision_float_cols)]
         for col in force_str_cols:
-            df[col] = df[col].astype(np.object).map(str_print).astype(np.str)
+            df[col] = df[col].map(str_print)
         for col in nanable_int_cols:
-            df[col] = df[col].astype(np.object).map(int_print).astype(np.str)
+            df[col] = df[col].map(int_print)
         for col in low_precision_float_cols:
             float_format = lambda f: ("{:.1g}" if f < 1 else "{:.1f}").format(f)
-            df[col] = df[col].astype(np.float).map(partial(num_print, float_format)).astype(np.float)
+            # The .astype(float) is required to maintain NaN as 'NaN' instead of 'nan'
+            df[col] = df[col].map(float_format).astype(float)
         for col in high_precision_float_cols:
-            df[col] = df[col].map(partial(num_print, "{:.6g}".format)).astype(np.float)
+            df[col] = df[col].map("{:.6g}".format).astype(float)
 
         cols = ([] if verbosity == 0
                 else ['task', 'fold', 'framework', 'constraint', 'result', 'metric', 'info'] if verbosity == 1
@@ -181,15 +190,17 @@ def as_printable_data_frame(self, verbosity=3):
                 else slice(None))
         return df.loc[:, cols]
 
-    def _load(self):
-        return self.load_df(self._score_file())
+    def load(self):
+        self.scores = self.load_df(self.path)
+        return self
 
     def save(self, append=False):
-        self.save_df(self.as_printable_data_frame(), path=self._score_file(), append=append)
+        self.save_df(self.as_printable_data_frame(), path=self.path, append=append)
+        return self
 
     def append(self, board_or_df, no_duplicates=True):
         to_append = board_or_df.as_data_frame() if isinstance(board_or_df, Scoreboard) else board_or_df
-        scores = self.as_data_frame().append(to_append, sort=False)
+        scores = self.as_data_frame().append(to_append)
         if no_duplicates:
             scores = scores.drop_duplicates()
         return Scoreboard(scores=scores,
@@ -198,7 +209,8 @@ def append(self, board_or_df, no_duplicates=True):
                           task_name=self.task_name,
                           scores_dir=self.scores_dir)
 
-    def _score_file(self):
+    @property
+    def path(self):
         sep = rconfig().token_separator
         if self.framework_name:
             if self.task_name:
@@ -228,12 +240,17 @@ def load_predictions(predictions_file):
             try:
                 df = read_csv(predictions_file, dtype=object)
                 log.debug("Predictions preview:\n %s\n", df.head(10).to_string())
+
                 if rconfig().test_mode:
                     TaskResult.validate_predictions(df)
-                if df.shape[1] > 2:
-                    return ClassificationResult(df)
+
+                if  'y_past_period_error' in df.columns:
+                    return TimeSeriesResult(df)
                 else:
-                    return RegressionResult(df)
+                    if df.shape[1] > 2:
+                        return ClassificationResult(df)
+                    else:
+                        return RegressionResult(df)
             except Exception as e:
                 return ErrorResult(ResultError(e))
         else:
@@ -254,6 +271,7 @@ def load_metadata(metadata_file):
     def save_predictions(dataset: Dataset, output_file: str,
                          predictions: Union[A, DF, S] = None, truth: Union[A, DF, S] = None,
                          probabilities: Union[A, DF] = None, probabilities_labels: Union[list, A] = None,
+                         optional_columns: Union[A, DF] = None,
                          target_is_encoded: bool = False,
                          preview: bool = True):
         """ Save class probabilities and predicted labels to file in csv format.
@@ -264,6 +282,7 @@ def save_predictions(dataset: Dataset, output_file: str,
         :param predictions:
         :param truth:
         :param probabilities_labels:
+        :param optional_columns:
         :param target_is_encoded:
         :param preview:
         :return: None
@@ -308,6 +327,10 @@ def save_predictions(dataset: Dataset, output_file: str,
 
         df = df.assign(predictions=preds)
         df = df.assign(truth=truth)
+
+        if optional_columns is not None:
+            df = pd.concat([df, optional_columns], axis=1)
+
         if preview:
             log.info("Predictions preview:\n %s\n", df.head(20).to_string())
         backup_file(output_file)
@@ -421,6 +444,11 @@ def compute_score(self, result=None, meta_result=None):
         required_meta_res = ['training_duration', 'predict_duration', 'models_count']
         for m in required_meta_res:
             entry[m] = meta_result[m] if m in meta_result else nan
+
+        if inference_times := Namespace.get(meta_result, "inference_times"):
+            for data_type, measurements in Namespace.dict(inference_times).items():
+                for n_samples, measured_times in Namespace.dict(measurements).items():
+                    entry[f"infer_batch_size_{data_type}_{n_samples}"] = np.median(measured_times)
         result = self.get_result() if result is None else result
 
         scoring_errors = []
@@ -450,7 +478,7 @@ def set_score(score):
         entry.info = result.info
         if scoring_errors:
             entry.info = "; ".join(filter(lambda it: it, [entry.info, *scoring_errors]))
-        entry |= Namespace({k: v for k, v in meta_result if k not in required_meta_res})
+        entry |= Namespace({k: v for k, v in meta_result if k not in required_meta_res and k != "inference_times"})
         log.info("Metric scores: %s", entry)
         return entry
 
@@ -656,6 +684,71 @@ def r2(self):
         """R^2"""
         return float(r2_score(self.truth, self.predictions))
 
+class TimeSeriesResult(RegressionResult):
+
+    def __init__(self, predictions_df, info=None):
+        super().__init__(predictions_df, info)
+        self.truth = self.df['truth'].values if self.df is not None else None #.iloc[:, 1].values if self.df is not None else None
+        self.predictions = self.df['predictions'].values if self.df is not None else None #.iloc[:, -2].values if self.df is not None else None
+        self.y_past_period_error = self.df['y_past_period_error'].values
+        self.quantiles = self.df.iloc[:, 2:-1].values
+        self.quantiles_probs = np.array([float(q) for q in self.df.columns[2:-1]])
+        self.truth = self.truth.astype(float, copy=False)
+        self.predictions = self.predictions.astype(float, copy=False)
+        self.quantiles = self.quantiles.astype(float, copy=False)
+        self.y_past_period_error = self.y_past_period_error.astype(float, copy=False)
+
+        self.target = Feature(0, 'target', 'real', is_target=True)
+        self.type = DatasetType.timeseries
+
+    @metric(higher_is_better=False)
+    def mase(self):
+        """Mean Absolute Scaled Error"""
+        return float(np.nanmean(np.abs(self.truth/self.y_past_period_error - self.predictions/self.y_past_period_error)))
+
+    @metric(higher_is_better=False)
+    def smape(self):
+        """Symmetric Mean Absolute Percentage Error"""
+        num = np.abs(self.truth - self.predictions)
+        denom = (np.abs(self.truth) + np.abs(self.predictions)) / 2
+        # If the denominator is 0, we set it to float('inf') such that any division yields 0 (this
+        # might not be fully mathematically correct, but at least we don't get NaNs)
+        denom[denom == 0] = math.inf
+        return np.mean(num / denom)
+
+    @metric(higher_is_better=False)
+    def mape(self):
+        """Symmetric Mean Absolute Percentage Error"""
+        num = np.abs(self.truth - self.predictions)
+        denom = np.abs(self.truth)
+        # If the denominator is 0, we set it to float('inf') such that any division yields 0 (this
+        # might not be fully mathematically correct, but at least we don't get NaNs)
+        denom[denom == 0] = math.inf
+        return np.mean(num / denom)
+
+    @metric(higher_is_better=False)
+    def nrmse(self):
+        """Normalized Root Mean Square Error"""
+        return self.rmse() / np.mean(np.abs(self.truth))
+
+    @metric(higher_is_better=False)
+    def wape(self):
+        """Weighted Average Percentage Error"""
+        return np.sum(np.abs(self.truth - self.predictions)) / np.sum(np.abs(self.truth))
+
+    @metric(higher_is_better=False)
+    def ncrps(self):
+        """Normalized Continuous Ranked Probability Score"""
+        quantile_losses = 2 * np.sum(
+            np.abs(
+                (self.quantiles - self.truth[:, None])
+                * ((self.quantiles >= self.truth[:, None]) - self.quantiles_probs[None, :])
+            ),
+            axis=0,
+        )
+        denom = np.sum(np.abs(self.truth)) # shape [num_time_series, num_quantiles]
+        weighted_losses = quantile_losses.sum(0) / denom  # shape [num_quantiles]
+        return weighted_losses.mean()
 
 _encode_predictions_and_truth_ = False
 
diff --git a/amlb/runners/aws.py b/amlb/runners/aws.py
index 4368efde3..6aae9cff1 100644
--- a/amlb/runners/aws.py
+++ b/amlb/runners/aws.py
@@ -13,6 +13,7 @@
 - merge downloaded results with existing/local results.
 - properly cleans up AWS resources (S3, EC2).
 """
+import datetime
 from concurrent.futures import ThreadPoolExecutor
 import copy as cp
 import datetime as dt
@@ -38,8 +39,9 @@
 from ..job import Job, JobError, MultiThreadingJobRunner, SimpleJobRunner, State as JobState
 from ..resources import config as rconfig, get as rget
 from ..results import ErrorResult, NoResultError, Scoreboard, TaskResult
-from ..utils import Namespace as ns, countdown, datetime_iso, file_filter, flatten, list_all_files, normalize_path, \
-    retry_after, retry_policy, str_def, str_iter, tail, touch
+from ..utils import Namespace as ns, countdown, datetime_iso, file_filter, flatten, \
+    list_all_files, normalize_path, \
+    retry_after, retry_policy, str_def, str_iter, tail, touch, Namespace
 from .docker import DockerBenchmark
 
 
@@ -196,7 +198,7 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
                 try:
                     jobs = flatten([self._make_aws_job([task_def.name], folds) for task_def in task_defs])
                     results = self._run_jobs(jobs)
-                    return self._process_results(results, task_name=tasks)
+                    return self._results_summary(self._process_results(results))
                 finally:
                     self.cleanup()
             else:
@@ -208,7 +210,8 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
                 task_names = None if tasks is None else [task_def.name for task_def in task_defs]
                 job = self._make_aws_job(task_names, folds)
                 results = self._run_jobs([job])
-                return self._process_results(results, task_name=tasks)
+                scoreboard = self._process_results(results)
+                return self._results_summary(scoreboard)
             finally:
                 self.cleanup()
 
@@ -217,7 +220,8 @@ def _create_job_runner(self, jobs):
             return SimpleJobRunner(jobs)
         else:
             queueing_strategy = MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority
-            return MultiThreadingJobRunner(jobs, self.parallel_jobs,
+            return MultiThreadingJobRunner(jobs,
+                                           parallel_jobs=self.parallel_jobs,
                                            delay_secs=rconfig().job_scheduler.delay_between_jobs,
                                            done_async=True,
                                            queueing_strategy=queueing_strategy)
@@ -391,12 +395,22 @@ def _on_state(_self, state):
                                 "please terminate it manually or restart it (after clearing its UserData) if you want to inspect the instance.",
                                 _self.ext.instance_id)
                 _self.ext.terminate = terminate
+                instance = self.instances.get(_self.ext.instance_id, {})
+                start_time = Namespace.get(instance, 'start_time', '')
+                stop_time = Namespace.get(instance, 'stop_time', '')
+                log_time = datetime.datetime.now(
+                    datetime.timezone.utc
+                ).strftime("%Y-%m-%dT%H:%M:%S")
                 if failure:
                     self._exec_send((lambda reason, **kwargs: self._save_failures(reason, **kwargs)),
                                     failure,
                                     tasks=_self.ext.tasks,
                                     folds=_self.ext.folds,
-                                    seed=_self.ext.seed)
+                                    seed=_self.ext.seed,
+                                    start_time=start_time,
+                                    stop_time=stop_time,
+                                    log_time=log_time,
+                                    )
 
             elif state == JobState.rescheduling:
                 self._stop_instance(_self.ext.instance_id, terminate=True, wait=False)
@@ -574,6 +588,8 @@ def _start_instance(self, instance_def, script_params="", instance_key=None, tim
             if instance_def.volume_size:
                 ebs['VolumeSize'] = instance_def.volume_size
 
+            instance_tags = ec2_config.instance_tags | ns(Name=f"amlb_{inst_key}")
+            volume_tags = (ec2_config.volume_tags or instance_tags) | ns(Name=f"amlb_{inst_key}")
             instance_params = dict(
                 BlockDeviceMappings=[dict(
                     DeviceName=ec2_config.root_device_name,
@@ -588,15 +604,11 @@ def _start_instance(self, instance_def, script_params="", instance_key=None, tim
                 TagSpecifications=[
                     dict(
                         ResourceType='instance',
-                        Tags=[
-                            dict(Key='Name', Value=f"benchmark_{inst_key}")
-                        ]
+                        Tags=[dict(Key=k, Value=v) for k, v in instance_tags]
                     ),
                     dict(
                         ResourceType='volume',
-                        Tags=[
-                            dict(Key='Name', Value=f"benchmark_{inst_key}")
-                        ]
+                        Tags=[dict(Key=k, Value=v) for k, v in volume_tags]
                     ),
                 ],
                 UserData=self._ec2_startup_script(inst_key, script_params=script_params, timeout_secs=timeout_secs)
@@ -744,8 +756,11 @@ def _save_failures(self, reason, **kwargs):
                         str_iter(kwargs.get('tasks', [])),
                         str_iter(kwargs.get('folds', [])),
                         str_def(kwargs.get('seed', None)),
+                        kwargs.get('start_time', "unknown"),
+                        kwargs.get('stop_time', "unknown"),
+                        kwargs.get('log_time', "unknown"),
                         str_def(reason, if_none="unknown"))],
-                      columns=['framework', 'benchmark', 'constraint', 'tasks', 'folds', 'seed', 'error'],
+                      columns=['framework', 'benchmark', 'constraint', 'tasks', 'folds', 'seed', 'start_time', 'stop_time', 'log_time', 'error'],
                       header=not os.path.exists(file),
                       path=file,
                       append=True)
@@ -902,10 +917,8 @@ def download_file(obj, dest, dest_display_path=None):
                 dest_path = os.path.join(self.output_dirs.session, rel_path)
                 try:
                     download_file(obj, dest_path)
-                    # if obj.key == result_key:
                     if is_result and not success:
-                        if rconfig().results.save:
-                            self._exec_send(lambda path: self._append(Scoreboard.load_df(path)), dest_path)
+                        self._exec_send(lambda path: self._save_global(Scoreboard.from_file(path)), dest_path)
                         success = True
                 except Exception as e:
                     if is_result:
@@ -926,6 +939,11 @@ def download_file(obj, dest, dest_display_path=None):
         self._update_instance(instance_id, success=success)
         return success, error
 
+    def _results_summary(self, scoreboard=None):
+        log.info(
+            "Result summary not available for AWS mode (reference files instead)."
+        )
+
     def _create_instance_profile(self):
         """
         see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
diff --git a/amlb/runners/container.py b/amlb/runners/container.py
index f33f220ab..c4a6110e0 100644
--- a/amlb/runners/container.py
+++ b/amlb/runners/container.py
@@ -93,7 +93,8 @@ def run(self, tasks: Union[str, List[str]] = None, folds: Union[int, List[int]]
             job = self._make_container_job(tasks, folds)
             try:
                 results = self._run_jobs([job])
-                return self._process_results(results, task_name=tasks)
+                scoreboard = self._process_results(results)
+                return self._results_summary(scoreboard)
             finally:
                 self.cleanup()
 
diff --git a/amlb/runners/docker.py b/amlb/runners/docker.py
index 2aef93bce..cf95bb153 100644
--- a/amlb/runners/docker.py
+++ b/amlb/runners/docker.py
@@ -9,7 +9,7 @@
 import re
 
 from ..benchmark import _setup_dir_
-from ..resources import config as rconfig, get as rget
+from ..resources import config as rconfig
 from ..utils import dir_of, run_cmd, str_digest, str_sanitize, touch
 from .container import ContainerBenchmark
 
@@ -50,15 +50,18 @@ def _start_container(self, script_params=""):
         custom_dir = rconfig().user_dir
         for d in [in_dir, out_dir, custom_dir]:
             touch(d, as_dir=True)
+
+        run_as = resolve_docker_run_as_option(rconfig().docker.run_as)
         script_extra_params = "--session="  # in combination with `self.output_dirs.session` usage below to prevent creation of 2 sessions locally
         inst_name = f"{self.sid}.{str_sanitize(str_digest(script_params))}"
         cmd = (
-            "docker run --name {name} {options} "
+            "docker run --name {name} {options} {run_as} "
             "-v {input}:/input -v {output}:/output -v {custom}:/custom "
             "--rm {image} {params} -i /input -o /output -u /custom -s skip -Xrun_mode=docker {extra_params}"
         ).format(
             name=inst_name,
             options=rconfig().docker.run_extra_options,
+            run_as=run_as,
             input=in_dir,
             output=self.output_dirs.session,
             custom=custom_dir,
@@ -94,12 +97,14 @@ def _image_exists(self, image):
 
     def _run_container_build_command(self, image, cache):
         log.info(f"Building docker image {image}.")
-        run_cmd("docker build {options} -t {container} -f {script} .".format(
+        run_cmd("docker build {options} {build_extra_options} -t {container} -f {script} .".format(
             options="" if cache else "--no-cache",
             container=image,
-            script=self._script),
+            script=self._script,
+            build_extra_options=rconfig().docker.build_extra_options,
+        ),
             _live_output_=rconfig().setup.live_output,
-            _activity_timeout_=rconfig().setup.activity_timeout
+            _activity_timeout_=rconfig().setup.activity_timeout,
         )
         log.info(f"Successfully built docker image {image}.")
 
@@ -109,18 +114,21 @@ def _upload_image(self, image):
         log.info(f"Successfully published docker image {image}.")
 
     def _generate_script(self, custom_commands):
-        docker_content = """FROM ubuntu:18.04
+        docker_content = """FROM ubuntu:22.04
 
 ENV DEBIAN_FRONTEND noninteractive
 RUN apt-get update
-RUN apt-get -y install apt-utils dialog locales
+RUN apt-get -y install apt-utils dialog locales tzdata sudo
 RUN apt-get -y install curl wget unzip git
 RUN apt-get -y install software-properties-common
 RUN add-apt-repository -y ppa:deadsnakes/ppa
 RUN apt-get update
 RUN apt-get -y install python{pyv} python{pyv}-venv python{pyv}-dev python3-pip
+RUN apt-get -y install libhdf5-serial-dev
 #RUN update-alternatives --install /usr/bin/python3 python3 $(which python{pyv}) 1
 
+RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
+
 # aliases for the python system
 ENV SPIP python{pyv} -m pip
 ENV SPY python{pyv}
@@ -132,6 +140,9 @@ def _generate_script(self, custom_commands):
 ENV LANG C.UTF-8
 ENV LC_ALL C.UTF-8
 
+# Setup HDF5 for installing `tables`
+ENV HDF5_DIR /usr/lib/aarch64-linux-gnu/hdf5/serial
+
 WORKDIR /bench
 
 # We create a virtual environment so that AutoML systems may use their preferred versions of
@@ -170,9 +181,34 @@ def _generate_script(self, custom_commands):
             pyv=rconfig().versions.python,
             pipv=rconfig().versions.pip,
             script=rconfig().script,
-            user=rconfig().user_dir,
         )
 
         touch(self._script)
         with open(self._script, 'w') as file:
             file.write(docker_content)
+
+
+def resolve_docker_run_as_option(option: str) -> str:
+    """ Resolve `docker.run_as` option into the correct `-u` option for `docker run`.
+
+    option, str: one of 'user' (unix only), 'root', 'default', or a valid `-u` option.
+               * 'user': set as `-u $(id -u):$(id -g)`, only on unix systems.
+               * 'root': set as `-u 0:0`
+               * 'default': does not set `-u`
+               * any string that starts with `-u`, which will be directly forwarded.
+
+    For linux specifically, files created within docker are *not always*
+    automatically owned by the user starting the docker instance.
+    We had reports of different behavior even for people running the same OS and Docker.
+    """
+    if option == "default":
+        return ''
+    if option == "root":
+        return '-u 0:0'
+    if option == "user":
+        if os.name == 'nt':
+            raise ValueError("docker.run_as: 'user' is not supported on Windows.")
+        return f'-u "{os.getuid()}:{os.getgid()}"'
+    if option.startswith("-u"):
+        return rconfig().docker.run_as
+    raise ValueError(f"Invalid setting for `docker.run_as`: '{option}'.")
diff --git a/amlb/runners/singularity.py b/amlb/runners/singularity.py
index 672db3770..451fccadb 100644
--- a/amlb/runners/singularity.py
+++ b/amlb/runners/singularity.py
@@ -155,7 +155,7 @@ def _upload_image(self, image):
 
     def _generate_script(self, custom_commands):
         singularity_content = """Bootstrap: docker
-From: ubuntu:18.04
+From: ubuntu:22.04
 %files
 . /bench/
 %post
diff --git a/amlb/uploads.py b/amlb/uploads.py
new file mode 100644
index 000000000..693500054
--- /dev/null
+++ b/amlb/uploads.py
@@ -0,0 +1,191 @@
+import json
+import logging
+import pathlib
+from collections import OrderedDict
+import textwrap
+from typing import Set, Optional, List
+
+import openml
+import pandas as pd
+from openml import OpenMLTask, OpenMLFlow
+from openml.runs.functions import format_prediction
+
+from .utils.core import Namespace
+
+log = logging.getLogger(__name__)
+
+
+def _load_task_data(task_folder: pathlib.Path, fold: int = 0) -> Namespace:
+    """ Loads the metadata of the given fold of a task as a namespace. """
+    with open(task_folder / str(fold) / 'metadata.json', 'r') as fh:
+        metadata = json.load(fh)
+    metadata = Namespace.from_dict(metadata)
+    return metadata
+
+
+def _load_fold(task_folder: pathlib.Path, fold: int, task: OpenMLTask) -> pd.DataFrame:
+    """ Load the predictions and add openml repeat/fold/index information. """
+    prediction_file = task_folder / f"{fold}/predictions.csv"
+    predictions = pd.read_csv(prediction_file, sep=',', header=0)
+
+    train_indices, test_indices = task.get_train_test_split_indices(fold, repeat=0, sample=0)
+    predictions["index"] = test_indices
+    predictions["fold"] = fold
+    predictions["repeat"] = 0
+    return predictions
+
+
+def _load_predictions(task_folder: pathlib.Path) -> pd.DataFrame:
+    """ Loads predictions of all folds for a task with index information required for upload. """
+    metadata = _load_task_data(task_folder)
+    task = openml.tasks.get_task(metadata.openml_task_id)
+    results = [_load_fold(task_folder, fold, task) for fold in range(10)]
+    return pd.concat(results)
+
+
+def _list_completed_folds(task_folder: pathlib.Path) -> Set[str]:
+    completed_folds = set()
+    for fold_dir in [d for d in task_folder.iterdir() if d.is_dir()]:
+        if (task_folder / fold_dir / "predictions.csv").is_file():
+            completed_folds.add(fold_dir.name)
+    return completed_folds
+
+
+def _get_flow(metadata: Namespace, sync_with_server: bool = True) -> openml.flows.OpenMLFlow:
+    """ Creates or retrieves an OpenML flow for the given run metadata. """
+    if metadata.git_info.repo == "NA":
+        msg = "Can't generate flow for results since repository information is unknown."
+        raise ValueError(msg)
+
+    amlb_version = next(iter(metadata.git_info.tags), metadata.git_info.commit[:8])
+
+    *_, owner, _ = metadata.git_info.repo.rsplit('/')
+    if owner != 'openml':
+        amlb_version = f"{owner}#{amlb_version}"
+
+    flow_description = """
+        This flow is generated by {fork}the automl benchmark: {repo}
+        Repository commit: {commit}
+        {framework_name} version: {framework_version}
+    """.format(
+        fork="a fork of " if owner != 'openml' else '',
+        repo=metadata.git_info.repo,
+        commit=metadata.git_info.commit,
+        framework_name=metadata.framework,
+        framework_version=metadata.framework_version,
+    )
+
+    amlb_flow = openml.flows.OpenMLFlow(
+        name=f"amlb_{metadata.framework}",
+        description=textwrap.dedent(flow_description),
+        external_version=f'amlb=={amlb_version},{metadata.framework}=={metadata.framework_version}',
+        # The values below are default values for a flow, the run will record used values.
+        parameters=OrderedDict(
+            max_runtime_seconds='14400',
+            max_mem_size_mb='32768',
+            cores='8',
+            seed='42',
+        ),
+        parameters_meta_info=OrderedDict(
+            max_runtime_seconds=dict(data_type='int', description='Maximum runtime in seconds.'),
+            max_mem_size_mb=dict(data_type='int', description='Memory constraint in megabytes.'),
+            cores=dict(data_type='int', description='Number of available cores.'),
+            seed=dict(data_type='int', description='The random seed.')
+        ),
+        language='English',
+        # We can use components to describe subflows, e.g. the automl framework with its hyperparameters.
+        # For now we don't.
+        components=OrderedDict(),
+        model=None,
+        tags=["amlb"],
+        dependencies=f'amlb=={amlb_version},{metadata.framework}=={metadata.framework_version}',
+    )
+    if sync_with_server:
+        # Publish will check if the flow exists on the server.
+        # If it exists, the local object is updated with server information.
+        # Otherwise the new flow is stored on the server.
+        return amlb_flow.publish()
+    else:
+        return amlb_flow
+
+
+def _extract_and_format_hyperparameter_configuration(metadata: Namespace, flow: OpenMLFlow) -> List[OrderedDict]:
+    return [
+        OrderedDict([
+            ("oml:name", name),
+            ("oml:value", metadata.__dict__.get(name, default)),
+            ("oml:component", flow.id)
+        ])
+        for name, default in flow.parameters.items()
+    ]
+
+
+def _upload_results(task_folder: pathlib.Path) -> openml.runs.OpenMLRun:
+    metadata = _load_task_data(task_folder)
+    predictions = _load_predictions(task_folder)
+
+    oml_flow = _get_flow(metadata)
+    oml_task = openml.tasks.get_task(metadata.openml_task_id)
+
+    denormalize_map = {label.strip().lower(): label for label in oml_task.class_labels}
+    predictions.columns = [col if col not in denormalize_map else denormalize_map[col] for col in predictions]
+    predictions["truth"] = predictions["truth"].apply(denormalize_map.get)
+    predictions["predictions"] = predictions["predictions"].apply(denormalize_map.get)
+
+    formatted_predictions = []
+    for _, row in predictions.iterrows():
+        if metadata.type != "classification":
+            class_probabilities = None
+        else:
+            class_probabilities = {c: row[c] for c in oml_task.class_labels}
+        prediction = format_prediction(
+               task=oml_task,
+               repeat=row["repeat"],
+               fold=row["fold"],
+               index=row["index"],
+               prediction=row["predictions"],
+               truth=row["truth"],
+               proba=class_probabilities,
+           )
+        formatted_predictions.append(prediction)
+
+    parameters = _extract_and_format_hyperparameter_configuration(metadata, oml_flow)
+    tags = ['amlb']
+    if 'tag' in metadata and metadata.tag not in [None, 'amlb']:
+        tags.extend([metadata.tag])
+
+    if len(metadata.framework_params) > 0:
+        description = "Framework_Hyperparameters-"
+        description += '-'.join(f"{hp}:{value}" for hp, value in Namespace.dict(metadata.framework_params).items())
+    else:
+        description = None
+
+    return openml.runs.OpenMLRun(
+        task_id=oml_task.id,
+        flow_id=oml_flow.id,
+        dataset_id=oml_task.dataset_id,
+        parameter_settings=parameters,
+        setup_string=metadata.command,
+        data_content=formatted_predictions,
+        tags=tags,
+        run_details=description,
+    ).publish()
+
+
+def missing_folds(task_folder: pathlib.Path) -> Set[str]:
+    completed_folds = _list_completed_folds(task_folder)
+    all_folds = {str(f) for f in range(10)}
+    return all_folds.difference(completed_folds)
+
+
+def process_task_folder(task_folder: pathlib.Path) -> Optional[openml.runs.OpenMLRun]:
+    """ Uploads the task folder iff predictions for all 10 folds are present. """
+    if len(missing_folds(task_folder)) > 0:
+        log.warning(
+            "Task %s is missing predictions for folds %s.",
+            task_folder,
+            ', '.join(missing_folds(task_folder))
+        )
+        return None
+
+    return _upload_results(task_folder)
diff --git a/amlb/utils/cache.py b/amlb/utils/cache.py
index 8c4282c69..00bc612a9 100644
--- a/amlb/utils/cache.py
+++ b/amlb/utils/cache.py
@@ -4,6 +4,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 _CACHE_PROP_PREFIX_ = '__cached__'
 
 
@@ -77,3 +79,4 @@ def decorator(self):
     return decorator
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/config.py b/amlb/utils/config.py
index 97b26ebc7..59bf7db30 100644
--- a/amlb/utils/config.py
+++ b/amlb/utils/config.py
@@ -1,42 +1,49 @@
 from collections import namedtuple
 from copy import deepcopy
 from dataclasses import dataclass
+from importlib.util import find_spec
 import logging
 import os
 from typing import Callable, List, Union
 
-from ruamel.yaml.constructor import SafeConstructor
-from ruamel.yaml.main import YAML
-
 from .core import Namespace, identity, json_load
 from .os import normalize_path
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+if find_spec('ruamel') is not None:
+    from ruamel.yaml.constructor import SafeConstructor
+    from ruamel.yaml.main import YAML
+    __no_export |= set(dir())
 
-class YAMLNamespaceConstructor(SafeConstructor):
+    class _YAMLNamespaceConstructor(SafeConstructor):
 
-    @classmethod
-    def init(cls):
-        cls.add_constructor(u'tag:yaml.org,2002:map', cls.construct_yaml_map)
+        @classmethod
+        def init(cls):
+            cls.add_constructor(u'tag:yaml.org,2002:map', cls.construct_yaml_map)
 
-    def construct_yaml_map(self, node):
-        data = Namespace()
-        yield data
-        value = self.construct_mapping(node)
-        data += value
+        def construct_yaml_map(self, node):
+            data = Namespace()
+            yield data
+            value = self.construct_mapping(node)
+            data += value
 
 
-YAMLNamespaceConstructor.init()
+    _YAMLNamespaceConstructor.init()
 
 
-def yaml_load(file, as_namespace=False):
-    if as_namespace:
-        yaml = YAML(typ='safe', pure=True)
-        yaml.Constructor = YAMLNamespaceConstructor
-    else:
-        yaml = YAML(typ='safe')
-    return yaml.load(file)
+    def yaml_load(file, as_namespace=False):
+        if as_namespace:
+            yaml = YAML(typ='safe', pure=True)
+            yaml.Constructor = _YAMLNamespaceConstructor
+        else:
+            yaml = YAML(typ='safe')
+        return yaml.load(file)
+else:
+    def yaml_load(*_, **__):
+        raise ImportError("ruamel.yaml package is required to load `yaml` config files.")
 
 
 def config_load(path, verbose=False):
@@ -87,3 +94,5 @@ def transform_config(config: Namespace, transform_rules: [TransformRule], inplac
                 Namespace.delete(config, k)
     return config
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/core.py b/amlb/utils/core.py
index 55469c9f6..e37f25984 100644
--- a/amlb/utils/core.py
+++ b/amlb/utils/core.py
@@ -11,22 +11,10 @@
 import re
 import sys
 import threading
-import types
 
 log = logging.getLogger(__name__)
 
-
-def register_module(module_name):
-    if module_name not in sys.modules:
-        mod = types.ModuleType(module_name)
-        sys.modules[module_name] = mod
-    return sys.modules[module_name]
-
-
-def register_submodule(mod, name):
-    fullname = '.'.join([mod.__name__, name])
-    module = register_module(fullname)
-    setattr(mod, name, module)
+__no_export = set(dir())  # all variables defined above this are not exported
 
 
 class Namespace:
@@ -234,11 +222,29 @@ def __json__(self):
         return Namespace.dict(self)
 
 
-def repr_def(obj, show_private=False):
+def _attributes(obj, filtr='all'):
+    attrs = vars(obj)
+    if filtr is None or filtr == 'all':
+        return attrs
+    elif filtr == 'public':
+        return {k: v for k, v in attrs.items() if not k.startswith('_')}
+    elif filtr == 'private':
+        return {k: v for k, v in attrs.items() if k.startswith('_')}
+    elif isinstance(filtr, list):
+        return {k: v for k, v in attrs.items() if k in filtr}
+    else:
+        assert callable(filtr)
+        return {k: v for k, v in attrs.items() if filtr(k)}
+
+
+def _classname(obj):
+    return type(obj).__name__
+
+
+def repr_def(obj, attributes='public'):
     return "{cls}({attrs!r})".format(
-        cls=type(obj).__name__,
-        attrs=(vars(obj) if show_private
-               else {k: v for k, v in vars(obj).items() if k.startswith('_')})
+        cls=_classname(obj),
+        attrs=_attributes(obj, attributes)
     )
 
 
@@ -460,3 +466,5 @@ def gen(*args, **kwargs):
         return threadsafe_iterator(fn(*args, **kwargs))
     return gen
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/modules.py b/amlb/utils/modules.py
index 0010409ff..4a2a1c4a6 100644
--- a/amlb/utils/modules.py
+++ b/amlb/utils/modules.py
@@ -1,4 +1,6 @@
 import logging
+import sys
+import types
 
 try:
     from pip._internal import main as pip_main
@@ -8,6 +10,21 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+
+def register_module(module_name):
+    if module_name not in sys.modules:
+        mod = types.ModuleType(module_name)
+        sys.modules[module_name] = mod
+    return sys.modules[module_name]
+
+
+def register_submodule(mod, name):
+    fullname = '.'.join([mod.__name__, name])
+    module = register_module(fullname)
+    setattr(mod, name, module)
+
 
 def pip_install(module_or_requirements, is_requirements=False):
     try:
@@ -19,3 +36,5 @@ def pip_install(module_or_requirements, is_requirements=False):
         log.error("Error when trying to install python modules %s.", module_or_requirements)
         log.exception(se)
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/os.py b/amlb/utils/os.py
index 39bd33f35..08c686a0b 100644
--- a/amlb/utils/os.py
+++ b/amlb/utils/os.py
@@ -12,6 +12,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def to_mb(size_in_bytes):
     return size_in_bytes / (1 << 20)
@@ -174,3 +176,4 @@ def __exit__(self, *args):
         # pass
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/process.py b/amlb/utils/process.py
index 593ed1ff8..d50d8615d 100644
--- a/amlb/utils/process.py
+++ b/amlb/utils/process.py
@@ -23,11 +23,30 @@
 import psutil
 
 from .core import Namespace, as_list, flatten, fn_name
-from .os import dir_of, to_mb
+from .os import dir_of, to_mb, path_from_split, split_path
 from .time import Timeout, Timer
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
+
+@contextmanager
+def file_lock(path, timeout=-1):
+    """
+    :param path: the path of the file to lock.
+        A matching lock file is automatically generated and associated to the file that we want to manipulate.
+    :param timeout: timeout in seconds to wait for the lock to be acquired. Disabled by default.
+    :raise: Timeout if the lock could not be acquired after timeout.
+    """
+    import filelock
+    splits = split_path(path)
+    splits.basename = f".{splits.basename}"  # keep the lock file as a hidden file as it's not deleted by filelock on release
+    splits.extension = f"{splits.extension}.lock"
+    lock_path = path_from_split(splits, real_path=False)
+    with filelock.FileLock(lock_path, timeout=timeout):
+        yield
+
 
 def run_subprocess(*popenargs,
                    input=None, capture_output=False, timeout=None, check=False, communicate_fn=None,
@@ -102,6 +121,8 @@ def live_output_windows(process: subprocess.Popen, **_) -> Tuple[str, str]:
     )
 
     def forward_output(stream, queue_):
+        if isinstance(stream, io.TextIOWrapper):
+            stream.reconfigure(errors="ignore")
         for line in stream:
             queue_.put(line)
 
@@ -151,9 +172,20 @@ def read_pipe(pipe, timeout):
                 reads[i] = line
         return reads if len(pipes) > 1 else reads[0]
 
-    output, error = zip(*iter(lambda: read_pipe([process.stdout if process.stdout else 1,
-                                                 process.stderr if process.stderr else 2], activity_timeout),
-                              ['', '']))
+    process_output = list(iter(
+        lambda: read_pipe(
+            [process.stdout if process.stdout else 1,
+             process.stderr if process.stderr else 2
+             ], activity_timeout
+    ), ['', '']))
+    if not process_output:
+        log.warning(
+            "No framework process output detected, "
+            "this might indicate a problem with the logging configuration."
+        )
+        return '', ''
+
+    output, error = zip(*process_output)
     print()  # ensure that the log buffer is flushed at the end
     return ''.join(output), ''.join(error)
 
@@ -741,3 +773,4 @@ def profiler(*args, **kwargs):
     return decorator
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/serialization.py b/amlb/utils/serialization.py
index 6b554855c..dffa4c53c 100644
--- a/amlb/utils/serialization.py
+++ b/amlb/utils/serialization.py
@@ -10,6 +10,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def _import_data_libraries():
     try:
@@ -236,3 +238,4 @@ def deserialize_data(path, config: Optional[ns] = None):
         raise SerializationError(f"Can not deserialize file `{path}` in unknown format.")
 
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/amlb/utils/time.py b/amlb/utils/time.py
index e73dcae6e..416688bcc 100644
--- a/amlb/utils/time.py
+++ b/amlb/utils/time.py
@@ -9,6 +9,8 @@
 
 log = logging.getLogger(__name__)
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def datetime_iso(datetime=None, date=True, time=True, micros=False, date_sep='-', datetime_sep='T', time_sep=':', micros_sep='.', no_sep=False):
     """
@@ -160,3 +162,5 @@ def __exit__(self, *args):
         if self.timer:
             self.timer.cancel()
 
+
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/docs/.nojekyll b/docs/.nojekyll
new file mode 100644
index 000000000..e69de29bb
diff --git a/docs/Gemfile b/docs/Gemfile
deleted file mode 100644
index 24c316eba..000000000
--- a/docs/Gemfile
+++ /dev/null
@@ -1,3 +0,0 @@
-source 'https://rubygems.org'
-gem 'github-pages', group: :jekyll_plugins
-gem "jekyll-theme-hydeout", "~> 4"
diff --git a/docs/HOWTO.md b/docs/HOWTO.md
index 4276a0880..de72a14ea 100644
--- a/docs/HOWTO.md
+++ b/docs/HOWTO.md
@@ -145,7 +145,7 @@ Each dataset must contain a training set and a test set. There can be multiple t
 
 ### Datasets definition
 
-A dataset definition consists in a `yaml` file listing all the task/datasets that will used for the complete benchmark, 
+A dataset definition consists in a `yaml` file listing all the task/datasets that will be used for the complete benchmark, 
 or as an OpenML suite.
 
 Default dataset definitions are available under folder `resources/benchmarks`.
@@ -474,7 +474,7 @@ Among the existing frameworks, we can see different type of integrations:
 - Python API integration: these are frameworks that can be run directly from Python: cf. `autosklearn`, `H2OAutoML`, `TPOT`, `RandomForest`, `TunedRandomForest`.
    - contrary to the trivial integration, those require a `setup` phase.
    - Most of them currently run using the same dependencies as the application, which is not recommended due to potential version conflicts (especially with `sklearn`). This was not a major constraint with the first frameworks implemented, but now, those integrations can and will be slightly changed to [run in their dedicated virtual environment](#frameworks-requiring-a-dedicated-virtual-env), using their own dependencies: cf. `RandomForest` and `examples/custom/extensions/Stacking` for examples.
-- non-Python frameworks: those frameworks typically run in `R` or `Java` and don't provide any Python API. The integration is then still done by spawning the `R` or `Java` process from the `exec.py`: cf. `AutoWEKA`, `autoxgboost`.
+- non-Python frameworks: those frameworks typically run in `R` or `Java` and don't provide any Python API. The integration is then still done by spawning the `Java` or `R` process from the `exec.py`: cf. `AutoWEKA` or `ranger`, respectively.
 
 #### Recommended structure
 
@@ -619,18 +619,21 @@ frameworks/ranger/
 
 Here are the main differences:
 - the `setup` phase is identical, but if at runtime, some executable file or library is required that need to be installed locally (as opposed to globally: for example, `R` or `java` executable are usually installed globally), we just recommend to put everything under the integration module (for example in `lib` and/or `bin` subfolders as for `AutoWEKA`). This is also true for some Python frameworks (cf. `hyperoptsklearn` integration for example, where the modules are loaded from `frameworks/hyperoptsklearn/lib/hyperopt-sklearn`).
-- the framework is then executed by building a command manually in `exec.py`, runming it in a subprocess, and finally collecting the results generated by the subprocess. For example, in `autoxgboost/exec.py`:
+- the framework is then executed by building a command manually in `exec.py`, running it in a subprocess, and finally collecting the results generated by the subprocess. For example, in `ranger/exec.py`:
   ```python
   with Timer() as training:
-      run_cmd(r"""Rscript --vanilla -e "source('{script}'); run('{train}', '{test}', target.index = {target_index}, '{output}', {cores}, time.budget = {time_budget})" """.format(
-          script=os.path.join(here, 'exec.R'),
-          train=dataset.train.path,
-          test=dataset.test.path,
-          target_index=dataset.target.index+1,
-          output=config.output_predictions_file,
-          cores=config.cores,
-          time_budget=config.max_runtime_seconds
-      ), _live_output_=True)
+    run_cmd(("Rscript --vanilla -e \""
+             "source('{script}'); "
+             "run('{train}', '{test}', '{output}', cores={cores}, meta_results_file='{meta_results}', task_type='{task_type}')"
+             "\"").format(
+        script=os.path.join(here, 'exec.R'),
+        train=dataset.train.path,
+        test=dataset.test.path,
+        output=config.output_predictions_file,
+        meta_results=meta_results_file,
+        task_type=config.type,
+        cores=config.cores
+    ), _live_output_=True)
   ```
   Here, the `exec.R` script is also responsible to save the predictions in the expected format.
 
diff --git a/docs/README.md b/docs/README.md
index dc4e676ef..491eeb627 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -38,7 +38,7 @@ Documentation: <https://openml.github.io/automlbenchmark/>
 ## Installation
 ### Pre-requisites
 To run the benchmarks, you will need:
-* Python 3.7+.
+* Python 3.9+.
 * PIP3: ensure you have a recent version. If necessary, upgrade your pip using `python -m pip install -U pip`.
 * The Python libraries listed in [requirements.txt](../requirements.txt): it is strongly recommended to first create a [Python virtual environment](https://docs.python.org/3/library/venv.html#venv-def) (cf. also [Pyenv](https://github.com/pyenv/pyenv): quick install using `curl https://pyenv.run | bash` or `brew install pyenv`) and work in it if you don't want to mess up your global Python environment.
 * [Docker](https://docs.docker.com/install/), if you plan to run the benchmarks in a container.
@@ -70,7 +70,7 @@ venv\Scripts\activate
 
 or using pyenv:
 ```bash
-pyenv install {python_version: 3.7.4}
+pyenv install {python_version: 3.9.16}
 pyenv virtualenv ve-automl
 pyenv local ve-automl
 ```
@@ -111,65 +111,74 @@ python3 runbenchmark.py --help
 ```text
 usage: runbenchmark.py [-h] [-m {local,aws,docker,singularity}]
                        [-t [task_id [task_id ...]]]
-                       [-f [fold_num [fold_num ...]]] [-i input_dir]
+                       [-f [fold_num ...]] [-i input_dir]
                        [-o output_dir] [-u user_dir] [-p parallel_jobs]
                        [-s {auto,skip,force,only}] [-k [true|false]]
+                       [-e] [--logging LOGGING]
+                       [--openml-run-tag OPENML_RUN_TAG]
                        framework [benchmark] [constraint]
 
 positional arguments:
-  framework             The framework to evaluate as defined by default in
-                        resources/frameworks.yaml. To use a labelled framework
-                        (i.e. a framework defined in
-                        resources/frameworks_{label}.yaml), use the syntax
-                        {framework}:{label}.
-  benchmark             The benchmark type to run as defined by default in
-                        resources/benchmarks/{benchmark}.yaml, a path to a
-                        benchmark description file, or an openml suite or
-                        task. OpenML references should be formatted as
-                        'openml/s/X' and 'openml/t/Y', for studies and tasks
-                        respectively. Defaults to `test`.
-  constraint            The constraint definition to use as defined by default
-                        in resources/constraints.yaml. Defaults to `test`.
+  framework             The framework to evaluate as defined by default in resources/frameworks.yaml.
+                        To use a labelled framework (i.e. a framework defined in resources/frameworks-{label}.yaml),
+                        use the syntax {framework}:{label}.
+  benchmark             The benchmark type to run as defined by default in resources/benchmarks/{benchmark}.yaml,
+                        a path to a benchmark description file, or an openml suite or task.
+                        OpenML references should be formatted as 'openml/s/X' and 'openml/t/Y',
+                        for studies and tasks respectively. Use 'test.openml/s/X' for the 
+                        OpenML test server.
+                        (default: 'test')
+  constraint            The constraint definition to use as defined by default in resources/constraints.yaml.
+                        (default: 'test')
 
 optional arguments:
   -h, --help            show this help message and exit
-  -m {local,docker,aws,singularity}, --mode {local,docker,aws,singularity}
-                        The mode that specifies how/where the benchmark tasks
-                        will be running. Defaults to local.
-  -t [task_id [task_id ...]], --task [task_id [task_id ...]]
-                        The specific task name (as defined in the benchmark
-                        file) to run. When an OpenML reference is used as
-                        benchmark, the dataset name should be used instead. If
-                        not provided, then all tasks from the benchmark will
-                        be run.
-  -f [fold_num [fold_num ...]], --fold [fold_num [fold_num ...]]
-                        If task is provided, the specific fold(s) to run. If
-                        fold is not provided, then all folds from the task
-                        definition will be run.
+  -m {local,aws,docker,singularity}, --mode {local,aws,docker,singularity}
+                        The mode that specifies how/where the benchmark tasks will be running.
+                        (default: 'local')
+  -t [task_id ...], --task [task_id ...]
+                        The specific task name (as defined in the benchmark file) to run.
+                        When an OpenML reference is used as benchmark, the dataset name should be used instead.
+                        If not provided, then all tasks from the benchmark will be run.
+  -f [fold_num ...], --fold [fold_num ...]
+                        If task is provided, the specific fold(s) to run.
+                        If fold is not provided, then all folds from the task definition will be run.
   -i input_dir, --indir input_dir
-                        Folder where datasets are loaded by default. Defaults
-                        to `input_dir` as defined in resources/config.yaml
+                        Folder from where the datasets are loaded by default.
+                        (default: '~/.openml')
   -o output_dir, --outdir output_dir
-                        Folder where all the outputs should be written.
-                        Defaults to `output_dir` as defined in
-                        resources/config.yaml
+                        Folder where all the outputs should be written.(default: './results')
   -u user_dir, --userdir user_dir
-                        Folder where all the customizations are stored.
-                        Defaults to `user_dir` as defined in
-                        resources/config.yaml
+                        Folder where all the customizations are stored.(default: '~/.config/automlbenchmark')
   -p parallel_jobs, --parallel parallel_jobs
-                        The number of jobs (i.e. tasks or folds) that can run
-                        in parallel. Defaults to 1. Currently supported only
-                        in docker and aws mode.
+                        The number of jobs (i.e. tasks or folds) that can run in parallel.
+                        A hard limit is defined by property `job_scheduler.max_parallel_jobs`
+                         in `resources/config.yaml`.
+                        Override this limit in your custom `config.yaml` file if needed.
+                        Supported only in aws mode or container mode (docker, singularity).
+                        (default: 1)
   -s {auto,skip,force,only}, --setup {auto,skip,force,only}
-                        Framework/platform setup mode. Defaults to auto.
-                        •auto: setup is executed only if strictly necessary.
-                        •skip: setup is skipped. •force: setup is always
-                        executed before the benchmark. •only: only setup is
-                        executed (no benchmark).
+                        Framework/platform setup mode. Available values are:
+                        • auto: setup is executed only if strictly necessary.
+                        • skip: setup is skipped.
+                        • force: setup is always executed before the benchmark.
+                        • only: only setup is executed (no benchmark).
+                        (default: 'auto')
   -k [true|false], --keep-scores [true|false]
-                        Set to true [default] to save/add scores in output
-                        directory.
+                        Set to true (default) to save/add scores in output directory.
+  -e, --exit-on-error   If set, terminates on the first task that does not complete with a model.
+  --logging LOGGING     Set the log levels for the 3 available loggers:
+                        • console
+                        • app: for the log file including only logs from amlb (.log extension).
+                        • root: for the log file including logs from libraries (.full.log extension).
+                        Accepted values for each logger are: notset, debug, info, warning, error, fatal, critical.
+                        Examples:
+                          --logging=info (applies the same level to all loggers)
+                          --logging=root:debug (keeps defaults for non-specified loggers)
+                          --logging=console:warning,app:info
+                        (default: 'console:info,app:debug,root:info')
+  --openml-run-tag OPENML_RUN_TAG
+                        Tag that will be saved in metadata and OpenML runs created during upload, must match '([a-zA-Z0-9_\-\.])+'.
 ```
 
 The script will produce output that records task metadata and the result.
@@ -327,6 +336,32 @@ By default, a benchmark run creates the following subdirectories and files in th
 * `logs`: this subdirectory contains logs produced by the `automlbenchmark` app, including when it's been run in Docker container or on AWS.
 
 
+### Uploading results to OpenML
+The `upload_results.py` script can be used to upload results to OpenML with the following usage:
+```text
+>python upload_results.py --help
+usage: Script to upload results from the benchmark to OpenML. [-h] [-i INPUT_DIRECTORY] [-a APIKEY] [-m MODE] [-x] [-v] [-t TASK]
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -i INPUT_DIRECTORY, --input-directory INPUT_DIRECTORY
+                        Directory that stores results from the runbenchmark.py invocation. By default use the most recent folder in the results folder as
+                        specified in the configuration.
+  -a APIKEY, --api-key APIKEY
+                        OpenML API key to use for uploading results.
+  -m MODE, --mode MODE  Run mode (default=check).
+                        • check: only report whether results can be uploaded.
+                        • upload: upload all complete results.
+  -x, --fail-fast       Stop as soon as a task fails to upload due to an error during uploading.
+  -v, --verbose         Output progress to console.
+  -t TASK, --task TASK  Only upload results for this specific task.
+```
+
+Note that the default behavior does not upload data but only verifies data is complete.
+We strongly encourage you to only upload your data after verifying all expected results are complete.
+The OpenML Python package is used for uploading results, so to ensure your API credentials are configured, please refer to their [configuration documentation](https://openml.github.io/openml-python/master/usage.html#installation-set-up).
+Results obtained on tasks on the test server (e.g. through the `--test-server` option of `runbenchmark.py`) are uploaded to the test server and don't require additional authentication.
+
 ## Advanced configuration
 If you need to create your own benchmark, add a framework, create a plugin for a proprietary framework, or simply want to use some advanced options (e.g. run some frameworks with non-default parameters), see the [HOWTO].
 
diff --git a/docs/_config.yml b/docs/_config.yml
deleted file mode 100644
index f5bc67e41..000000000
--- a/docs/_config.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-repository: openml/automlbenchmark
-#theme: jekyll-theme-slate
-remote_theme: fongandrew/hydeout
-markdown: kramdown
-
-title: AutoML Benchmark
-#description: config description
-
-#sidebar_home_link: true
-version: 0.9
-
-hydeout:
-  repo: https://github.com/openml/automlbenchmark
-  version: 0.9
-
-defaults:
-  -
-    scope:
-      path: README.md
-    values:
-      title: ''
-  -
-    scope:
-      path: HOWTO.md
-    values:
-      title: ''
diff --git a/docs/assets/css/main.scss b/docs/assets/css/main.scss
deleted file mode 100644
index 61a51042f..000000000
--- a/docs/assets/css/main.scss
+++ /dev/null
@@ -1,16 +0,0 @@
----
-# Jekyll needs front matter for SCSS files
-# $sidebar-bg-color: #ac4142;
-# $sidebar-sticky: false;
----
-
-$sidebar-width: 20rem !default;
-$large-font-size: 1.1rem;
-$link-color: #195cc6;
-@import "hydeout";
-
-#sidebar {
-    .site-title {
-		font-family: Sans-Serif;
-    }
-}
\ No newline at end of file
diff --git a/docs/automl_overview.html b/docs/automl_overview.html
new file mode 100644
index 000000000..e72f07c04
--- /dev/null
+++ b/docs/automl_overview.html
@@ -0,0 +1,17 @@
+<html>
+  <head>
+    <title>AMLB</title>
+    <meta charset="UTF-8" />
+    <meta http-equiv="refresh" content="3; URL=frameworks.html" />
+  </head>
+  <body>
+    <p>
+      The old AutoML framework overview page no longer exists, you will be
+      redirected to the new page. If you are not redirected within 3 seconds,
+      you can find it at
+      <a href="https://openml.github.io/automlbenchmark/frameworks.html"
+        >https://openml.github.io/automlbenchmark/frameworks.html</a
+      >.
+    </p>
+  </body>
+</html>
diff --git a/docs/benchmark_datasets.html b/docs/benchmark_datasets.html
new file mode 100644
index 000000000..f9b28623f
--- /dev/null
+++ b/docs/benchmark_datasets.html
@@ -0,0 +1,18 @@
+<html>
+  <head>
+    <title>AMLB</title>
+    <meta charset="UTF-8" />
+    <meta http-equiv="refresh" content="5; URL=https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md"/>
+  </head>
+  <body>
+    <p>
+      You probably found this link from our 2019 paper.
+      Unfortunately, we updated our site but the new website does not yet contain a page with a description of our dataset selection strategy.<br/>
+        You will be redirected to the markdown file from which the old page was generated, so you can still view the old text.<br/>
+        If you are not redirected within 5 seconds, you can find it at
+      <a href="https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md"
+        >https://github.com/openml/automlbenchmark/blob/2fe3bd41768ce28387f827791bd57ef1a5a84783/docs/benchmark_datasets.md</a
+      >.
+    </p>
+  </body>
+</html>
diff --git a/docs/frameworks.html b/docs/frameworks.html
new file mode 100644
index 000000000..98f48c9d4
--- /dev/null
+++ b/docs/frameworks.html
@@ -0,0 +1,989 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Frameworks</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <h1 class="page-title">AutoML Frameworks</h1>
+        There is more to an AutoML system than just its performance. This page
+        contains more information about the integrated AutoML frameworks,
+        including links to their papers, repositories
+        <img src="img/logos/GitHub-Mark-64px.png" height="16px" />, and
+        documentation 📖. Summaries taken directly from the respective
+        documentation pages. Want to integrate your own framework?
+        <a
+          href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-an-automl-framework"
+          target="_blank"
+          >Adding your own framework</a
+        >
+        is relatively simple.
+        <!-- 
+        <br />
+        <br />
+        An AutoML framework may only be available through an API for a specific
+        programming language, while others can work stand-alone. Some systems
+        might output models which can be used without further dependency on the
+        AutoML package, in other cases the AutoML system is still required to
+        use the model. Some systems might be developed with a specific domain in
+        mind. When choosing an AutoML system, it is essential to consider things
+        that are important to you. -->
+      </section>
+      <section class="framework-card-list">
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/autogluon.png" height="32px" />
+            <h3>AutoGluon</h3>
+            <div class="framework-links">
+              <a href="https://github.com/awslabs/autogluon" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://auto.gluon.ai/stable/index.html" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            AutoGluon enables easy-to-use and easy-to-extend AutoML with a focus
+            on automated stack ensembling, deep learning, and real-world
+            applications spanning image, text, and tabular data.
+          </div>
+          <input type="checkbox" id="more-autogluon" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                AutoGluon-Tabular: Robust and Accurate AutoML for Structured
+                Data
+              </h3>
+              <div class="paper-authors">
+                Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang,
+                Pedro Larroy, Mu Li, Alexander Smola
+              </div>
+              <div class="paper-abstract">
+                We introduce AutoGluon-Tabular, an open-source AutoML framework
+                that requires only a single line of Python to train highly
+                accurate machine learning models on an unprocessed tabular
+                dataset such as a CSV file. Unlike existing AutoML frameworks
+                that primarily focus on model/hyperparameter selection,
+                AutoGluon-Tabular succeeds by ensembling multiple models and
+                stacking them in multiple layers. Experiments reveal that our
+                multi-layer combination of many models offers better use of
+                allocated training time than seeking out the best. A second
+                contribution is an extensive evaluation of public and commercial
+                AutoML platforms including TPOT, H2O, AutoWEKA, auto-sklearn,
+                AutoGluon, and Google AutoML Tables. Tests on a suite of 50
+                classification and regression tasks from Kaggle and the OpenML
+                AutoML Benchmark reveal that AutoGluon is faster, more robust,
+                and much more accurate. We find that AutoGluon often even
+                outperforms the best-in-hindsight combination of all of its
+                competitors. In two popular Kaggle competitions, AutoGluon beat
+                99% of the participating data scientists after merely 4h of
+                training on the raw data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2020</strong>
+                  <div>ICML 2020 AutoML Workshop</div>
+                </div>
+                <a href="https://arxiv.org/pdf/2003.06505.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a href="https://arxiv.org/abs/2003.06505" target="_blank"
+                  >arxiv</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-autogluon">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/auto-sklearn.png" height="32px" />
+            <h3>Auto-sklearn</h3>
+            <div class="framework-links">
+              <a href="https://github.com/automl/auto-sklearn" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://automl.github.io/auto-sklearn/master/#"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            Auto-sklearn is an automated machine learning toolkit and a drop-in
+            replacement for a scikit-learn estimator. Auto-sklearn frees a
+            machine learning user from algorithm selection and hyperparameter
+            tuning. It leverages recent advantages in Bayesian optimization,
+            meta-learning and ensemble construction.
+          </div>
+          <input type="checkbox" id="more-askl" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
+              </h3>
+              <div class="paper-authors">
+                Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius
+                Lindauer, Frank Hutter
+              </div>
+              <div class="paper-abstract">
+                Automated Machine Learning (AutoML) supports practitioners and
+                researchers with the tedious task of designing machine learning
+                pipelines and has recently achieved substantial success. In this
+                paper we introduce new AutoML approaches motivated by our
+                winning submission to the second ChaLearn AutoML challenge. We
+                develop PoSH Auto-sklearn, which enables AutoML systems to work
+                well on large datasets under rigid time limits using a new,
+                simple and meta-feature-free meta-learning technique and employs
+                a successful bandit strategy for budget allocation. However,
+                PoSH Auto-sklearn introduces even more ways of running AutoML
+                and might make it harder for users to set it up correctly.
+                Therefore, we also go one step further and study the design
+                space of AutoML itself, proposing a solution towards truly
+                hands-free AutoML. Together, these changes give rise to the next
+                generation of our AutoML system, Auto-sklearn 2.0 . We verify
+                the improvements by these additions in a large experimental
+                study on 39 AutoML benchmark datasets and conclude the paper by
+                comparing to other popular AutoML frameworks and Auto-sklearn
+                1.0 , reducing the relative error by up to a factor of 4.5, and
+                yielding a performance in 10 minutes that is substantially
+                better than what Auto-sklearn 1.0 achieves within an hour.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>arXiv</div>
+                </div>
+                <a href="https://arxiv.org/pdf/2007.04074.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a href="https://arxiv.org/abs/2007.04074" target="_blank"
+                  >arxiv</a
+                >
+              </div>
+            </div>
+            <div class="paper">
+              <h3 class="paper-title">
+                Efficient and Robust Automated Machine Learning
+              </h3>
+              <div class="paper-authors">
+                Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost
+                Springenberg, Manuel Blum, Frank Hutter
+              </div>
+              <div class="paper-abstract">
+                The success of machine learning in a broad range of applications
+                has led to an ever-growing demand for machine learning systems
+                that can be used off the shelf by non-experts. To be effective
+                in practice, such systems need to automatically choose a good
+                algorithm and feature preprocessing steps for a new dataset at
+                hand, and also set their respective hyperparameters. Recent work
+                has started to tackle this automated machine learning (AutoML)
+                problem with the help of efficient Bayesian optimization
+                methods. In this work we introduce a robust new AutoML system
+                based on scikit-learn (using 15 classifiers, 14 feature
+                preprocessing methods, and 4 data preprocessing methods, giving
+                rise to a structured hypothesis space with 110 hyperparameters).
+                This system, which we dub auto-sklearn, improves on existing
+                AutoML methods by automatically taking into account past
+                performance on similar datasets, and by constructing ensembles
+                from the models evaluated during the optimization. Our system
+                won the first phase of the ongoing ChaLearn AutoML challenge,
+                and our comprehensive analysis on over 100 diverse datasets
+                shows that it substantially outperforms the previous state of
+                the art in AutoML. We also demonstrate the performance gains due
+                to each of our contributions and derive insights into the
+                effectiveness of the individual components of auto-sklearn.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2015</strong>
+                  <div>
+                    Advances in Neural Information Processing Systems 28 (NIPS
+                    2015)
+                  </div>
+                </div>
+                <a
+                  href="https://papers.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://papers.neurips.cc/paper/2015/hash/11d0e6287202fced83f79975ec59a3a6-Abstract.html"
+                  target="_blank"
+                  >neurips</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-askl">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/flaml.svg" height="32px" />
+            <h3>FLAML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/microsoft/FLAML" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://microsoft.github.io/FLAML/" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            FLAML is a lightweight Python library that finds accurate machine
+            learning models automatically, efficiently and economically. It
+            frees users from selecting learners and hyperparameters for each
+            learner.
+          </div>
+          <input type="checkbox" id="more-flaml" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                FLAML: A Fast and Lightweight AutoML Library
+              </h3>
+              <div class="paper-authors">
+                Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu
+              </div>
+              <div class="paper-abstract">
+                We study the problem of using low computational cost to automate
+                the choices of learners and hyperparameters for an ad-hoc
+                training dataset and error metric, by conducting trials of
+                different configurations on the given training data. We
+                investigate the joint impact of multiple factors on both trial
+                cost and model error, and propose several design guidelines.
+                Following them, we build a fast and lightweight library FLAML
+                which optimizes for low computational resource in finding
+                accurate models. FLAML integrates several simple but effective
+                search strategies into an adaptive system. It significantly
+                outperforms top-ranked AutoML libraries on a large open source
+                AutoML benchmark under equal, or sometimes orders of magnitude
+                smaller budget constraints.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>MLSys Conference 2021</div>
+                </div>
+                <a
+                  href="https://proceedings.mlsys.org/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-flaml">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/GAMA-icon.png" height="32px" />
+            <h3>GAMA</h3>
+            <div class="framework-links">
+              <a href="https://github.com/openml-labs/gama" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://openml-labs.github.io/gama/master/"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            GAMA is developed for AutoML research and features a flexible AutoML
+            pipeline, which makes it easy to develop and evaluate new AutoML
+            components. GAMA's benchmarking configuration features evolutionary
+            optimization and ensemble construction.
+          </div>
+          <input type="checkbox" id="more-gama" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                GAMA: A General Automated Machine Learning Assistant
+              </h3>
+              <div class="paper-authors">
+                Pieter Gijsbers, Joaquin Vanschoren
+              </div>
+
+              <!-- <div class="paper-year">2019</div>
+            <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+            <div class="paper-venue">cite</div> -->
+
+              <div class="paper-abstract">
+                The General Automated Machine learning Assistant (GAMA) is a
+                modular AutoML system developed to empower users to track and
+                control how AutoML algorithms search for optimal machine
+                learning pipelines, and facilitate AutoML research itself. In
+                contrast to current, often black-box systems, GAMA allows users
+                to plug in different AutoML and post-processing techniques, logs
+                and visualizes the search process, and supports easy
+                benchmarking. It currently features three AutoML search
+                algorithms, two model post-processing steps, and is designed to
+                allow for more components to be added.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2021</strong>
+                  <div>
+                    Joint European Conference on Machine Learning and Knowledge
+                    Discovery in Databases. Springer, Cham, 2020.
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/2007.04911.pdf" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-030-67670-4_39"
+                  target="_blank"
+                  >Springer</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-gama">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/h2o-automl-logo.jpeg" height="32px" />
+            <h3>H2O AutoML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/h2oai/h2o-3" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            H2O's AutoML can be used for automating the machine learning
+            workflow, which includes automatic training and tuning of many
+            models within a user-specified time-limit. H2O offers a number of
+            model explainability methods that apply to AutoML objects (groups of
+            models), as well as individual models (e.g. leader model).
+            Explanations can be generated automatically with a single function
+            call, providing a simple interface to exploring and explaining the
+            AutoML models.
+          </div>
+          <input type="checkbox" id="more-h2o" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                H2O AutoML: Scalable Automatic Machine Learning
+              </h3>
+              <div class="paper-authors">Erin LeDell and Sébastien Poirier</div>
+              <div class="paper-abstract">
+                H2O is an open source, distributed machine learning platform
+                designed to scale to very large datasets, with APIs in R,
+                Python, Java and Scala. We present H2O AutoML, a highly
+                scalable, fully-automated, supervised learning algorithm which
+                automates the pro- cess of training a large selection of
+                candidate models and stacked ensembles within a single function.
+                The result of the AutoML run is a “leaderboard”: a ranked list
+                of models, all of which can be easily exported for use in a
+                production environment. Models in the leader- board can be
+                ranked by numerous model performance metrics or other model
+                attributes such as training time or average per-row prediction
+                speed. The H2O AutoML algorithm relies on the efficient training
+                of H2O machine learning al- gorithms to produce a large number
+                of models in a short amount of time. H2O AutoML uses a
+                combination of fast random search and stacked ensembles to
+                achieve results competitive with, and often better than, other
+                frameworks which rely on more complex model tuning techniques
+                such as Bayesian optimization or genetic algorithms. H2O AutoML
+                trains a va- riety of algorithms (e.g. GBMs, Random Forests,
+                Deep Neural Networks, GLMs), yielding a healthy amount of
+                diversity across candidate models, which can be exploited by
+                stacked ensembles to produce a powerful final model. The
+                effectiveness of this technique is reflected in the OpenML
+                AutoML Benchmark, which compares the performance of several of
+                the most well known, open source AutoML systems across a number
+                of datasets.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2020</strong>
+                  <div>ICML 2020 AutoML Workshop</div>
+                </div>
+                <a
+                  href="https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf"
+                  target="_blank"
+                  >PDF</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-h2o">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/LightAutoML_logo_small.png" height="32px" />
+            <h3>LightAutoML</h3>
+            <div class="framework-links">
+              <a href="https://github.com/sb-ai-lab/LightAutoML" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a
+                href="https://lightautoml.readthedocs.io/en/latest/"
+                target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            LightAutoML is open-source Python library aimed at automated machine
+            learning. It is designed to be lightweight and efficient for various
+            tasks with tabular, text data.
+          </div>
+          <input type="checkbox" id="more-lama" class="accordion-input" />
+          <div class="accordion-content">
+            <div>Paper to be added.</div>
+            <div class="paper">
+              <!-- <h3 class="paper-title">
+                Automating biomedical data science through tree-based pipeline
+                optimization
+              </h3> -->
+              <div class="paper-authors">
+                Alexander Ryzhkov, Anton Vakhrushev, Dmitry Simakov, Vasilii
+                Bunakov, Rinchin Damdinov, Alexander Kirilin, Pavel Shvets
+              </div>
+              <!-- <div class="paper-abstract">
+                Automated machine learning (AutoML) systems are helpful data
+                science assistants designed to scan data for novel features,
+                select appropriate supervised learning models and optimize their
+                parameters. For this purpose, Tree-based Pipeline Optimization
+                Tool (TPOT) was developed using strongly typed genetic
+                programing (GP) to recommend an optimized analysis pipeline for
+                the data scientist’s prediction problem. However, like other
+                AutoML systems, TPOT may reach computational resource limits
+                when working on big data such as whole-genome expression data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2016</strong>
+                  <div>
+                    Applications of Evolutionary Computation, pages 123-137
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/1601.07925" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-319-31204-0_9"
+                  target="_blank"
+                  >springer</a
+                >
+              </div> -->
+            </div>
+          </div>
+          <label for="more-lama">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/mljar.png" height="32px" />
+            <h3>mljar-supervised</h3>
+            <div class="framework-links">
+              <a
+                href="https://github.com/mljar/mljar-supervised/"
+                target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="https://supervised.mljar.com" target="_blank">📖</a>
+            </div>
+          </div>
+          <div>
+            The mljar-supervised is an Automated Machine Learning Python package
+            that works with tabular data. It is designed to save time for a data
+            scientist. It abstracts the common way to preprocess the data,
+            construct the machine learning models, and perform hyper-parameters
+            tuning to find the best model 🏆. It is no black-box as you can see
+            exactly how the ML pipeline is constructed (with a detailed Markdown
+            report for each ML model).
+          </div>
+          <input type="checkbox" id="more-mljar" class="accordion-input" />
+          <div class="accordion-content">
+            <div>No paper available.</div>
+          </div>
+          <label for="more-mljar">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+        <div class="accordion acard">
+          <div class="framework-header">
+            <img src="img/logos/tpot.jpeg" height="32px" />
+            <h3>TPOT</h3>
+            <div class="framework-links">
+              <a href="https://github.com/EpistasisLab/tpot" target="_blank"
+                ><img src="img/logos/GitHub-Mark-64px.png" height="24px"
+              /></a>
+              <a href="http://epistasislab.github.io/tpot/" target="_blank"
+                >📖</a
+              >
+            </div>
+          </div>
+          <div>
+            TPOT is a Python Automated Machine Learning tool that optimizes
+            machine learning pipelines using genetic programming. It has a focus
+            on optimizing models for biomedical data.
+          </div>
+          <input type="checkbox" id="more-tpot" class="accordion-input" />
+          <div class="accordion-content">
+            <div class="paper">
+              <h3 class="paper-title">
+                Automating biomedical data science through tree-based pipeline
+                optimization
+              </h3>
+              <div class="paper-authors">
+                Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A.
+                Lavender, La Creis Kidd, and Jason H. Moore
+              </div>
+              <div class="paper-abstract">
+                Automated machine learning (AutoML) systems are helpful data
+                science assistants designed to scan data for novel features,
+                select appropriate supervised learning models and optimize their
+                parameters. For this purpose, Tree-based Pipeline Optimization
+                Tool (TPOT) was developed using strongly typed genetic
+                programing (GP) to recommend an optimized analysis pipeline for
+                the data scientist’s prediction problem. However, like other
+                AutoML systems, TPOT may reach computational resource limits
+                when working on big data such as whole-genome expression data.
+              </div>
+              <div class="paper-links">
+                <div class="hover-expand">
+                  <strong>2016</strong>
+                  <div>
+                    Applications of Evolutionary Computation, pages 123-137
+                  </div>
+                </div>
+                <a href="https://arxiv.org/pdf/1601.07925" target="_blank"
+                  >PDF</a
+                >
+                <a
+                  href="https://link.springer.com/chapter/10.1007/978-3-319-31204-0_9"
+                  target="_blank"
+                  >springer</a
+                >
+              </div>
+            </div>
+          </div>
+          <label for="more-tpot">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/img/logos/GAMA-icon.png b/docs/img/logos/GAMA-icon.png
new file mode 100644
index 000000000..c9626aa28
Binary files /dev/null and b/docs/img/logos/GAMA-icon.png differ
diff --git a/docs/img/logos/GAMA.png b/docs/img/logos/GAMA.png
new file mode 100644
index 000000000..0ba100f0c
Binary files /dev/null and b/docs/img/logos/GAMA.png differ
diff --git a/docs/img/logos/GitHub-Mark-32px.png b/docs/img/logos/GitHub-Mark-32px.png
new file mode 100644
index 000000000..8b25551a9
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-32px.png differ
diff --git a/docs/img/logos/GitHub-Mark-64px.png b/docs/img/logos/GitHub-Mark-64px.png
new file mode 100644
index 000000000..182a1a3f7
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-64px.png differ
diff --git a/docs/img/logos/GitHub-Mark-Light-64px.png b/docs/img/logos/GitHub-Mark-Light-64px.png
new file mode 100644
index 000000000..73db1f61f
Binary files /dev/null and b/docs/img/logos/GitHub-Mark-Light-64px.png differ
diff --git a/docs/img/logos/LightAutoML_logo_small.png b/docs/img/logos/LightAutoML_logo_small.png
new file mode 100644
index 000000000..8d268e390
Binary files /dev/null and b/docs/img/logos/LightAutoML_logo_small.png differ
diff --git a/docs/img/logos/auto-sklearn.png b/docs/img/logos/auto-sklearn.png
new file mode 100644
index 000000000..65141af67
Binary files /dev/null and b/docs/img/logos/auto-sklearn.png differ
diff --git a/docs/img/logos/autogluon.png b/docs/img/logos/autogluon.png
new file mode 100644
index 000000000..8afef59ab
Binary files /dev/null and b/docs/img/logos/autogluon.png differ
diff --git a/docs/img/logos/flaml.svg b/docs/img/logos/flaml.svg
new file mode 100644
index 000000000..5ae22b683
--- /dev/null
+++ b/docs/img/logos/flaml.svg
@@ -0,0 +1,1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1138.16 450.71"><defs><style>.cls-1{fill:#ff9406;}.cls-2{fill:#505d66;}</style></defs><g id="Layer_2" data-name="Layer 2"><g id="图层_1" data-name="图层 1"><path class="cls-1" d="M298,76.7,173.81.24a1.58,1.58,0,0,0-2.06,2.35L211.55,51a263.57,263.57,0,0,0-33.37,3.9,252.77,252.77,0,0,0-35,9,240.65,240.65,0,0,0-33,13.92,228.41,228.41,0,0,0-30.5,18.8,211.86,211.86,0,0,0-29,25.52,191.34,191.34,0,0,0-23,29.72,176.8,176.8,0,0,0-16.34,33.49,172.22,172.22,0,0,0-8.9,36.76L0,241a1.58,1.58,0,0,0,1.37,1.77,1.6,1.6,0,0,0,1-.22l79-47.9a1.55,1.55,0,0,0,.69-.86l1-3.16a145,145,0,0,1,26.41-47.86,170.28,170.28,0,0,1,41.5-36A196.9,196.9,0,0,1,203.12,84.1a214.83,214.83,0,0,1,59.32-7.55c2.86,0,5.77.12,8.63.27s5.76.34,8.62.6,5.75.57,8.6.93,5.71.78,8.54,1.25a1.58,1.58,0,0,0,1.91-1.16A1.59,1.59,0,0,0,298,76.7Z" transform="translate(0.01 0.01)"/><path class="cls-1" d="M347.83,177.83l-1-18.05a1.57,1.57,0,0,0-1.65-1.48,1.49,1.49,0,0,0-.79.26l-71.16,47.15a1.55,1.55,0,0,0-.67,1L271.9,210a143.76,143.76,0,0,1-22.58,52.27,174.42,174.42,0,0,1-42.61,43,205,205,0,0,1-58.31,28.88,217.42,217.42,0,0,1-68.28,9.93c-3.3-.05-6.63-.17-9.89-.36s-6.58-.47-9.83-.8-6.51-.76-9.73-1.24-6.42-1.05-9.6-1.68a1.57,1.57,0,0,0-1.3,2.76L171.15,450.34a1.57,1.57,0,0,0,2.39-1.94l-36.87-70.52a264,264,0,0,0,40.57-5.5A251.22,251.22,0,0,0,217.75,360a238,238,0,0,0,36.9-18.61,224.15,224.15,0,0,0,32.27-24.25,201.9,201.9,0,0,0,28.2-31.37,179.69,179.69,0,0,0,19.59-34.43A167,167,0,0,0,345.6,215,161.86,161.86,0,0,0,347.83,177.83Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M258.56,209.79,196.9,181.24l61.42-95.48a1.63,1.63,0,0,0-2.23-2.26L101.25,179.84a1.63,1.63,0,0,0-.52,2.25,1.56,1.56,0,0,0,.67.6l60.26,29.12-90.33,122a1.62,1.62,0,0,0,.13,2.09,1.6,1.6,0,0,0,2.08.23l185.24-123.5a1.63,1.63,0,0,0,.46-2.26,1.67,1.67,0,0,0-.68-.58Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M451.86,199a36.63,36.63,0,0,0-12.3,10.44,32.45,32.45,0,0,0-6.35,14.49l-4.09,24.49h104.4l-4,24H425.12l-8.88,53.25H380.3l3-17.79,14-84a60.11,60.11,0,0,1,11.49-26.51,67.08,67.08,0,0,1,22.41-19.21,58.25,58.25,0,0,1,27.69-7.07H546.4l-4,24H466.76A31.6,31.6,0,0,0,451.86,199Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M772.68,325.65,742.8,208.39l-46.3,78,60.5,2.68L637.67,325.65,729,171.09h39.76l39.83,154.56Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M643.32,301.39H597.81q-11.91,0-18.91-8.43t-5-20.33l16.93-101.54H554.61L537.69,272.63q-2.4,14.39,2.65,26.51a41.67,41.67,0,0,0,16.11,19.32,45.49,45.49,0,0,0,25.42,7.19H629Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M975.05,170.86h36.17l-25.8,154.79H949.25L966,225l-37.44,45.37H892.87l-23.7-46.5-17,101.77H816l25.8-154.79H878l37.1,72.57Z" transform="translate(0.01 0.01)"/><path class="cls-2" d="M1138.15,301.39h-75.68q-11.91,0-18.92-8.43t-5-20.33l16.92-101.54H1019.3l-16.93,101.54a47.91,47.91,0,0,0,2.63,26.51,41.7,41.7,0,0,0,16.1,19.32,45.57,45.57,0,0,0,25.42,7.19h77.29Z" transform="translate(0.01 0.01)"/></g></g></svg>
\ No newline at end of file
diff --git a/docs/img/logos/h2o-automl-logo.jpeg b/docs/img/logos/h2o-automl-logo.jpeg
new file mode 100644
index 000000000..68ab95b32
Binary files /dev/null and b/docs/img/logos/h2o-automl-logo.jpeg differ
diff --git a/docs/img/logos/mljar.png b/docs/img/logos/mljar.png
new file mode 100644
index 000000000..a61427bef
Binary files /dev/null and b/docs/img/logos/mljar.png differ
diff --git a/docs/img/logos/tpot.jpeg b/docs/img/logos/tpot.jpeg
new file mode 100644
index 000000000..fd4d56add
Binary files /dev/null and b/docs/img/logos/tpot.jpeg differ
diff --git a/docs/img/shiny.png b/docs/img/shiny.png
new file mode 100644
index 000000000..9942dfeec
Binary files /dev/null and b/docs/img/shiny.png differ
diff --git a/docs/index.html b/docs/index.html
new file mode 100644
index 000000000..c6f1a808d
--- /dev/null
+++ b/docs/index.html
@@ -0,0 +1,443 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: an AutoML Benchmark</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <div class="title">AMLB</div>
+        <div class="subtitle">An AutoML Benchmark</div>
+        <div class="summary">
+          Comparing different AutoML frameworks is notoriously challenging. AMLB
+          is an open and extensible benchmark that follows best practices and
+          avoids common mistakes when comparing AutoML frameworks.
+        </div>
+      </section>
+      <section>
+        <div class="three-cols">
+          <div class="card">
+            <h3>Easy to Use</h3>
+            <div class="grey">
+              You can run an entire benchmark with a single command! The AutoML
+              benchmark tool automates the installation of the AutoML framework,
+              the experimental setup, and executing the experiment.
+            </div>
+            <div class="terminal">
+              > python runbenchmark.py autosklearn openml/s/269 1h8c
+            </div>
+            <div class="flex-grow"></div>
+            <a
+              class="card-nav"
+              href="https://github.com/openml/automlbenchmark#installation"
+              target="_blank"
+            >
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Installation guide
+            </a>
+          </div>
+          <div class="card">
+            <h3>Visualize Results</h3>
+            <div class="grey">
+              The results can be visualized with our
+              <a
+                href="https://compstat-lmu.shinyapps.io/AutoML-Benchmark-Analysis/"
+                target="_blank"
+                >interactive visualization tool</a
+              >
+              or one of our
+              <a
+                href="https://github.com/openml/automlbenchmark/tree/master/reports"
+                >notebooks</a
+              >. This includes stripplots, critical difference diagrams,
+              Bradley-Terry trees, and more!
+            </div>
+            <img src="img/shiny.png" width="100%" />
+            <div class="flex-grow"></div>
+            <a class="card-nav" href="results.html" target="_blank">
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Results
+            </a>
+          </div>
+          <div class="card">
+            <h3>Easy to Extend</h3>
+            <div class="grey">
+              <a
+                href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-a-benchmark"
+                >Adding a framework</a
+              >
+              and
+              <a
+                href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-a-benchmark"
+                >adding a dataset</a
+              >
+              is easy. These extensions can be kept completely private, or
+              shared with the community. For datasets, it is even possible to
+              work with
+              <a href="https://www.openml.org" target="_blank">OpenML</a>
+              tasks and suites directly!
+            </div>
+            <svg width="72" height="72" viewBox="0, 0, 400,400">
+              <g>
+                <path
+                  id="path0"
+                  d="M77.077 264.657 C 76.925 264.809,76.800 265.251,76.800 265.639 C 76.800 266.066,76.445 266.395,75.900 266.472 C 75.017 266.598,74.998 266.717,74.890 272.700 L 74.779 278.800 75.790 278.800 C 76.496 278.800,76.800 279.017,76.800 279.520 C 76.800 280.994,77.584 281.200,83.200 281.200 C 88.816 281.200,89.600 280.994,89.600 279.520 C 89.600 279.017,89.904 278.800,90.610 278.800 L 91.621 278.800 91.510 272.700 C 91.403 266.769,91.376 266.597,90.525 266.475 C 89.979 266.398,89.602 266.021,89.525 265.475 C 89.399 264.591,77.899 263.834,77.077 264.657 M89.867 302.667 C 89.720 302.813,89.600 303.251,89.600 303.639 C 89.600 304.066,89.245 304.395,88.700 304.472 C 87.806 304.599,87.800 304.645,87.800 311.600 C 87.800 318.555,87.806 318.601,88.700 318.728 C 89.283 318.811,89.600 319.131,89.600 319.638 C 89.600 320.862,91.193 321.200,96.926 321.195 C 103.118 321.188,103.852 321.043,104.174 319.762 C 104.321 319.176,104.707 318.800,105.161 318.800 C 106.215 318.800,106.388 317.773,106.395 311.474 C 106.400 305.959,106.091 304.400,104.994 304.400 C 104.675 304.400,104.302 303.950,104.164 303.400 L 103.913 302.400 97.023 302.400 C 93.234 302.400,90.013 302.520,89.867 302.667 M281.200 322.140 C 281.200 322.984,280.968 323.128,279.226 323.359 C 277.588 323.576,277.231 323.772,277.126 324.510 C 277.046 325.075,276.675 325.446,276.110 325.526 C 275.372 325.631,275.176 325.988,274.959 327.626 C 274.728 329.368,274.584 329.600,273.740 329.600 L 272.784 329.600 272.892 337.900 C 272.998 346.023,273.018 346.203,273.851 346.321 C 274.543 346.420,274.751 346.809,274.962 348.395 C 275.176 350.012,275.373 350.369,276.110 350.474 C 276.675 350.554,277.046 350.925,277.126 351.490 C 277.231 352.228,277.588 352.424,279.226 352.641 C 280.968 352.872,281.200 353.016,281.200 353.860 L 281.200 354.816 289.500 354.708 C 297.766 354.600,297.801 354.596,297.928 353.700 C 298.019 353.056,298.325 352.800,299.002 352.800 C 300.639 352.800,302.000 352.078,302.000 351.210 C 302.000 350.663,302.263 350.400,302.810 350.400 C 303.678 350.400,304.400 349.039,304.400 347.402 C 304.400 346.725,304.656 346.419,305.300 346.328 C 306.682 346.132,306.729 329.875,305.349 329.679 C 304.655 329.580,304.449 329.191,304.235 327.579 C 304.017 325.933,303.834 325.600,303.149 325.600 C 302.651 325.600,302.239 325.271,302.108 324.769 C 301.875 323.878,300.533 323.200,299.002 323.200 C 298.325 323.200,298.019 322.944,297.928 322.300 C 297.801 321.404,297.766 321.400,289.500 321.292 L 281.200 321.184 281.200 322.140 M117.067 334.267 C 116.920 334.413,116.800 334.863,116.800 335.267 C 116.800 335.782,116.503 336.000,115.800 336.000 C 115.008 336.000,114.800 336.202,114.800 336.972 C 114.800 337.676,114.551 337.980,113.900 338.072 C 113.010 338.199,112.999 338.280,112.891 345.290 L 112.782 352.380 113.768 352.628 C 114.371 352.779,114.802 353.210,114.877 353.738 C 114.952 354.265,115.335 354.648,115.862 354.723 C 116.390 354.798,116.821 355.229,116.972 355.832 L 117.220 356.818 124.310 356.709 C 131.320 356.601,131.401 356.590,131.528 355.700 C 131.620 355.049,131.924 354.800,132.628 354.800 C 133.391 354.800,133.600 354.591,133.600 353.828 C 133.600 353.124,133.849 352.820,134.500 352.728 C 135.391 352.601,135.401 352.528,135.509 345.300 L 135.618 338.000 134.609 338.000 C 133.799 338.000,133.600 337.802,133.600 337.000 C 133.600 336.200,133.400 336.000,132.600 336.000 C 131.800 336.000,131.600 335.800,131.600 335.000 L 131.600 334.000 124.467 334.000 C 120.543 334.000,117.213 334.120,117.067 334.267 M245.680 348.880 C 245.416 349.144,245.200 349.684,245.200 350.080 C 245.200 350.542,244.903 350.800,244.369 350.800 C 243.308 350.800,241.200 352.890,241.200 353.941 C 241.200 354.470,240.892 354.788,240.300 354.872 C 239.400 355.000,239.400 355.001,239.400 363.200 C 239.400 371.399,239.400 371.400,240.300 371.528 C 240.892 371.612,241.200 371.930,241.200 372.459 C 241.200 373.510,243.308 375.600,244.369 375.600 C 244.903 375.600,245.200 375.858,245.200 376.320 C 245.200 377.855,245.910 378.000,253.447 378.000 C 261.546 378.000,262.000 377.923,262.000 376.557 C 262.000 375.773,262.205 375.600,263.132 375.600 C 263.931 375.600,264.423 375.305,264.800 374.600 C 265.094 374.050,265.575 373.600,265.868 373.600 C 266.165 373.600,266.400 373.170,266.400 372.628 C 266.400 371.924,266.649 371.620,267.300 371.528 C 268.200 371.400,268.200 371.399,268.200 363.200 C 268.200 355.001,268.200 355.000,267.300 354.872 C 266.749 354.794,266.400 354.466,266.400 354.028 C 266.400 353.634,265.950 352.974,265.400 352.561 C 264.850 352.148,264.400 351.583,264.400 351.305 C 264.400 351.011,263.900 350.800,263.200 350.800 C 262.201 350.800,262.000 350.640,262.000 349.843 C 262.000 349.316,261.715 348.776,261.367 348.643 C 260.116 348.163,246.187 348.373,245.680 348.880 M155.288 353.272 C 155.020 353.540,154.800 353.981,154.800 354.252 C 154.800 354.523,154.395 354.802,153.900 354.872 C 153.262 354.963,152.964 355.313,152.876 356.075 C 152.789 356.828,152.489 357.188,151.876 357.275 C 150.474 357.475,150.498 373.329,151.900 373.528 C 152.551 373.620,152.800 373.924,152.800 374.628 C 152.800 375.386,153.010 375.600,153.757 375.600 C 154.430 375.600,154.784 375.881,154.951 376.549 C 155.266 377.803,156.387 378.000,163.200 378.000 C 170.013 378.000,171.134 377.803,171.449 376.549 C 171.616 375.881,171.970 375.600,172.643 375.600 C 173.390 375.600,173.600 375.386,173.600 374.628 C 173.600 373.924,173.849 373.620,174.500 373.528 C 175.394 373.401,175.401 373.351,175.508 365.493 L 175.617 357.587 174.608 357.203 C 173.898 356.933,173.600 356.521,173.600 355.810 C 173.600 355.028,173.396 354.800,172.695 354.800 C 172.140
+ 354.800,171.637 354.452,171.395 353.900 C 170.923 352.824,156.306 352.254,155.288 353.272 M203.476 357.058 C 203.324 357.209,203.200 357.743,203.200 358.245 C 203.200 359.048,202.975 359.171,201.300 359.278 L 199.400 359.400 199.279 361.275 C 199.169 362.979,199.059 363.162,198.079 363.275 L 197.000 363.400 197.000 370.600 L 197.000 377.800 198.079 377.925 C 199.059 378.038,199.169 378.221,199.279 379.925 L 199.400 381.800 201.275 381.921 C 202.979 382.031,203.162 382.141,203.275 383.121 L 203.400 384.200 210.600 384.200 L 217.800 384.200 217.925 383.121 C 218.038 382.141,218.221 382.031,219.925 381.921 L 221.800 381.800 221.921 379.925 C 222.031 378.221,222.141 378.038,223.121 377.925 L 224.200 377.800 224.200 370.600 L 224.200 363.400 223.121 363.275 C 222.141 363.162,222.031 362.979,221.921 361.275 L 221.800 359.400 219.925 359.279 C 218.221 359.169,218.038 359.059,217.925 358.079 L 217.800 357.000 210.776 356.891 C 206.912 356.831,203.627 356.906,203.476 357.058 "
+                  stroke="none"
+                  fill="#34cc64"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path1"
+                  d="M203.467 47.867 C 203.320 48.013,203.200 48.451,203.200 48.839 C 203.200 49.266,202.845 49.595,202.300 49.672 C 201.417 49.798,201.398 49.917,201.290 55.900 L 201.179 62.000 202.190 62.000 C 202.799 62.000,203.200 62.227,203.200 62.571 C 203.200 64.074,204.305 64.400,209.396 64.400 C 215.031 64.400,215.600 64.270,215.600 62.984 C 215.600 62.266,215.851 62.024,216.700 61.927 L 217.800 61.800 217.871 55.884 L 217.942 49.968 216.771 49.734 C 215.871 49.554,215.600 49.280,215.600 48.550 L 215.600 47.600 209.667 47.600 C 206.403 47.600,203.613 47.720,203.467 47.867 M176.276 87.858 C 176.124 88.009,176.000 88.463,176.000 88.867 C 176.000 89.270,175.820 89.623,175.600 89.650 C 175.380 89.678,174.975 89.723,174.700 89.750 C 174.324 89.788,174.181 91.551,174.125 96.863 L 174.050 103.926 175.025 104.171 C 175.910 104.393,176.062 104.604,175.970 105.484 C 175.903 106.130,177.702 106.344,183.200 106.344 C 188.698 106.344,190.497 106.130,190.430 105.484 C 190.338 104.605,190.490 104.393,191.365 104.173 C 193.344 103.676,193.472 89.799,191.500 89.626 C 191.335 89.612,191.033 89.600,190.828 89.600 C 190.623 89.600,190.398 89.195,190.328 88.700 C 190.199 87.794,177.123 87.010,176.276 87.858 M157.308 125.736 C 157.017 126.193,156.949 126.410,156.850 127.200 C 156.822 127.420,156.350 127.600,155.800 127.600 C 155.028 127.600,154.800 127.806,154.800 128.505 C 154.800 129.060,154.452 129.563,153.900 129.805 C 153.023 130.190,152.997 130.378,152.891 137.090 L 152.781 143.980 153.791 144.233 C 154.420 144.391,154.800 144.769,154.800 145.236 C 154.800 145.694,155.175 146.079,155.765 146.227 C 156.344 146.372,156.829 146.860,156.975 147.444 L 157.220 148.419 164.110 148.309 C 170.822 148.203,171.010 148.177,171.395 147.300 C 171.637 146.748,172.140 146.400,172.695 146.400 C 173.394 146.400,173.600 146.172,173.600 145.400 C 173.600 144.850,173.780 144.378,174.000 144.350 C 174.220 144.322,174.625 144.278,174.900 144.250 C 175.275 144.212,175.427 142.382,175.509 136.900 L 175.618 129.600 174.609 129.600 C 173.799 129.600,173.600 129.402,173.600 128.600 C 173.600 127.800,173.400 127.600,172.600 127.600 C 171.812 127.600,171.600 127.397,171.600 126.643 C 171.600 126.115,171.298 125.570,170.928 125.428 C 169.703 124.958,157.629 125.231,157.308 125.736 M146.892 174.607 C 146.612 174.925,146.393 175.323,146.404 175.492 C 146.415 175.661,145.969 175.857,145.412 175.927 C 144.583 176.030,144.400 176.269,144.400 177.250 C 144.400 177.952,144.233 178.344,143.997 178.198 C 142.725 177.412,142.400 179.078,142.400 186.400 C 142.400 193.722,142.725 195.388,143.997 194.602 C 144.219 194.465,144.400 194.687,144.400 195.096 C 144.400 196.048,145.152 196.800,146.104 196.800 C 146.513 196.800,146.735 196.981,146.598 197.203 C 145.812 198.475,147.478 198.800,154.800 198.800 C 162.122 198.800,163.788 198.475,163.002 197.203 C 162.859 196.972,163.194 196.800,163.786 196.800 C 164.614 196.800,164.868 196.562,165.059 195.606 C 165.190 194.950,165.457 194.512,165.652 194.632 C 166.873 195.387,167.200 193.646,167.200 186.400 C 167.200 179.078,166.875 177.412,165.603 178.198 C 165.367 178.344,165.200 177.952,165.200 177.250 C 165.200 176.242,165.027 176.033,164.100 175.927 C 163.495 175.857,163.051 175.659,163.114 175.486 C 163.671 173.954,148.206 173.116,146.892 174.607 M148.564 219.000 C 148.387 219.708,148.042 220.000,147.383 220.000 C 145.156 220.000,144.400 220.657,144.400 222.590 C 144.400 224.254,144.319 224.400,143.400 224.400 L 142.400 224.400 142.400 231.600 L 142.400 238.800 143.400 238.800 C 144.276 238.800,144.400 238.973,144.400 240.200 C 144.400 242.031,144.860 243.148,145.600 243.114 C 147.635 243.022,148.354 243.269,148.472 244.100 C 148.599 244.991,148.672 245.001,155.900 245.109 L 163.200 245.218 163.200 244.209 C 163.200 243.366,163.397 243.192,164.400 243.150 C 165.060 243.123,165.780 243.055,166.000 243.000 C 166.220 242.945,166.545 242.878,166.722 242.850 C 166.899 242.822,167.088 242.048,167.142 241.129 C 167.261 239.114,167.428 238.800,168.381 238.800 C 169.477 238.800,169.600 238.075,169.600 231.600 C 169.600 224.910,169.498 224.400,168.157 224.400 C 167.309 224.400,167.200 224.220,167.200 222.823 C 167.200 221.016,166.160 220.000,164.311 220.000 C 163.393 220.000,163.200 219.827,163.200 219.000 L 163.200 218.000 156.008 218.000 L 148.815 218.000 148.564 219.000 M168.100 258.265 C 167.825 258.376,167.600 258.812,167.600 259.233 C 167.600 259.740,167.321 260.000,166.780 260.000 C 165.687 260.000,163.200 262.459,163.200 263.540 C 163.200 264.117,162.951 264.400,162.443 264.400 C 162.026 264.400,161.576 264.685,161.443 265.033 C 161.019 266.136,161.165 280.796,161.600 280.850 C 161.820 280.877,162.223 280.923,162.495 280.950 C 162.767 280.978,163.056 281.360,163.138 281.800 C 163.327 282.821,163.995 283.650,164.418 283.389 C 164.598 283.278,164.875 283.398,165.034 283.655 C 165.215 283.948,165.150 284.016,164.862 283.838 C 164.215 283.439,164.285 283.942,164.971 284.629 C 165.286 284.943,166.006 285.200,166.571 285.200 C 167.402 285.200,167.600 285.393,167.600 286.200 L 167.600 287.200 175.576 287.200 C 183.535 287.200,184.174 287.096,184.194 285.800 C 184.197 285.580,184.695 285.343,185.300 285.273 C 185.905 285.204,186.400 284.955,186.400 284.720 C 186.400 284.485,186.833 283.912,187.362 283.447 C 187.891 282.981,188.388 282.240,188.467 281.800 C 188.546 281.360,188.788 280.997,189.005 280.994 C 190.292 280.974,190.400 280.323,190.400 272.584 L 190.400 264.815 189.400 264.564 C 188.793 264.412,188.400 264.028,188.400 263.587 C 188.400 262.718,186.246 260.474,184.931 259.974 C 184.416 259.778,184.017 259.479,184.044 259.309 C 184.250 258.046,183.974 258.000,176.300 258.032 C 172.065 258.049,168.375 258.154,168.100 258.265 M201.200 282.148 C 201.200 283.108,200.906 283.235,197.700 283.666 C 197.425 283.703,197.284 283.817,197.386 283.919 C 197.790 284.323,196.726 285.600,195.986 285.600 C 195.554 285.600,195.178 285.780,195.150 286.000 C 195.123 286.220,195.055 286.580,195.000 286.800 C 194.945 287.020,194.878 287.740,194.850 288.400 C 194.808 289.403,194.634 289.600,193.792 289.600 L 192.784 289.600 192.892 297.900 C 193.000 306.166,193.004 306.201,193.900 306.328 C 194.585 306.425,194.812 306.736,194.850 307.628 C 194.878 308.273,194.945 308.980,195.000 309.200 C 195.055 309.420,195.123 309.791,195.150 310.024 C 195.178 310.257,195.373 310.340,195.585 310.209 C 196.048 309.923,197.714 311.492,197.432 311.948 C 197.324 312.123,197.452 312.297,197.718 312.334 C 200.960 312.786,201.200 312.890,201.200 313.852 L 201.200 314.816 209.500 314.708 C 217.766 314.600,217.801 314.596,217.928 313.700 C 218.020 313.054,218.325 312.800,219.009 312.800 C 220.477 312.800,222.000 311.953,222.000 311.136 C 222.000 310.688,222.288 310.400,222.736 310.400 C 223.559 310.400,224.400 308.874,224.400 307.381 C 224.400 306.842,224.580 306.377,224.800 306.350 C 225.020 306.322,225.425 306.278,225.700 306.250 C 226.736 306.146,226.382 289.826,225.341 289.678 C 224.521 289.561,224.336 289.090,223.934 286.100 C 223.897 285.825,223.783 285.684,223.681 285.786 C 223.356 286.110,222.000 285.194,222.000 284.650 C 222.000 284.016,220.316 283.200,219.009 283.200 C 218.325 283.200,218.020 282.946,217.928 282.300 C 217.801 281.404,217.766 281.400,209.500 281.292 L 201.200 281.184 201.200 282.148 "
+                  stroke="none"
+                  fill="#fccc04"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path2"
+                  d="M289.877 3.456 C 289.725 3.609,289.600 4.051,289.600 4.439 C 289.600 4.866,289.245 5.195,288.700 5.272 C 287.811 5.398,287.800 5.475,287.800 11.600 C 287.800 17.725,287.811 17.802,288.700 17.928 C 289.351 18.020,289.600 18.324,289.600 19.028 L 289.600 20.000 295.800 20.000 L 302.000 20.000 302.000 19.027 C 302.000 18.262,302.236 18.026,303.100 17.927 L 304.200 17.800 304.200 11.600 L 304.200 5.400 303.125 5.276 C 302.372 5.189,302.012 4.889,301.925 4.276 C 301.799 3.391,290.698 2.636,289.877 3.456 M256.276 31.058 C 256.124 31.209,256.000 31.651,256.000 32.039 C 256.000 32.466,255.645 32.795,255.100 32.872 C 254.206 32.999,254.200 33.045,254.200 40.000 C 254.200 46.955,254.206 47.001,255.100 47.128 C 255.737 47.218,256.000 47.526,256.000 48.181 C 256.000 49.438,256.775 49.589,263.200 49.589 C 269.659 49.589,270.400 49.441,270.400 48.153 C 270.400 47.416,270.616 47.200,271.353 47.200 C 272.644 47.200,272.788 46.467,272.795 39.874 C 272.800 34.138,272.501 32.800,271.214 32.800 C 270.732 32.800,270.409 32.472,270.328 31.900 C 270.199 30.994,257.123 30.210,256.276 31.058 M228.676 64.658 C 228.524 64.809,228.400 65.263,228.400 65.667 C 228.400 66.182,228.103 66.400,227.400 66.400 C 226.608 66.400,226.400 66.602,226.400 67.372 C 226.400 68.076,226.151 68.380,225.500 68.472 C 224.605 68.599,224.600 68.638,224.600 75.800 C 224.600 82.962,224.605 83.001,225.500 83.128 C 226.151 83.220,226.400 83.524,226.400 84.228 C 226.400 84.991,226.609 85.200,227.372 85.200 C 228.076 85.200,228.380 85.449,228.472 86.100 C 228.599 86.991,228.672 87.001,235.900 87.109 L 243.200 87.218 243.200 86.209 C 243.200 85.399,243.398 85.200,244.200 85.200 C 245.000 85.200,245.200 85.000,245.200 84.200 C 245.200 83.398,245.399 83.200,246.209 83.200 L 247.218 83.200 247.109 75.900 C 247.001 68.672,246.991 68.599,246.100 68.472 C 245.449 68.380,245.200 68.076,245.200 67.372 C 245.200 66.609,244.991 66.400,244.228 66.400 C 243.524 66.400,243.220 66.151,243.128 65.500 C 242.999 64.595,229.524 63.809,228.676 64.658 M203.600 103.400 C 203.333 104.102,202.919 104.400,202.210 104.400 C 201.412 104.400,201.200 104.600,201.200 105.357 C 201.200 106.043,200.918 106.384,200.200 106.564 L 199.200 106.815 199.200 114.800 L 199.200 122.785 200.200 123.036 C 200.918 123.216,201.200 123.557,201.200 124.243 C 201.200 125.000,201.412 125.200,202.210 125.200 C 202.919 125.200,203.333 125.498,203.600 126.200 L 203.980 127.200 211.782 127.200 L 219.585 127.200 219.836 126.200 C 220.016 125.482,220.357 125.200,221.043 125.200 C 221.797 125.200,222.000 124.988,222.000 124.200 C 222.000 123.463,222.213 123.200,222.810 123.200 C 224.102 123.200,224.400 121.627,224.400 114.800 C 224.400 107.973,224.102 106.400,222.810 106.400 C 222.213 106.400,222.000 106.137,222.000 105.400 C 222.000 104.612,221.797 104.400,221.043 104.400 C 220.357 104.400,220.016 104.118,219.836 103.400 L 219.585 102.400 211.782 102.400 L 203.980 102.400 203.600 103.400 M188.564 147.400 C 188.399 148.057,188.034 148.400,187.500 148.400 C 186.381 148.400,184.400 150.570,184.400 151.796 C 184.400 152.602,184.202 152.800,183.392 152.800 L 182.384 152.800 182.492 161.100 C 182.600 169.366,182.604 169.401,183.500 169.528 C 184.157 169.621,184.400 169.924,184.400 170.646 C 184.400 171.921,186.079 173.600,187.354 173.600 C 188.076 173.600,188.379 173.843,188.472 174.500 C 188.599 175.396,188.634 175.400,196.900 175.508 L 205.200 175.616 205.200 174.608 C 205.200 173.826,205.404 173.600,206.111 173.600 C 207.261 173.600,209.600 171.499,209.600 170.466 C 209.600 169.967,209.962 169.596,210.608 169.434 L 211.617 169.181 211.508 161.090 C 211.401 153.042,211.395 152.999,210.500 152.872 C 209.904 152.788,209.600 152.470,209.600 151.932 C 209.600 150.867,207.114 148.400,206.040 148.400 C 205.411 148.400,205.200 148.149,205.200 147.400 L 205.200 146.400 197.008 146.400 L 188.815 146.400 188.564 147.400 M188.667 191.067 C 188.520 191.213,188.400 191.663,188.400 192.067 C 188.400 192.570,188.105 192.800,187.458 192.800 C 186.186 192.800,184.400 194.524,184.400 195.753 C 184.400 196.477,184.157 196.779,183.500 196.872 C 182.604 196.999,182.600 197.034,182.492 205.300 L 182.384 213.600 183.392 213.600 C 184.202 213.600,184.400 213.798,184.400 214.604 C 184.400 215.830,186.381 218.000,187.500 218.000 C 188.034 218.000,188.399 218.343,188.564 219.000 L 188.815 220.000 197.008 220.000 L 205.200 220.000 205.200 219.000 C 205.200 218.251,205.411 218.000,206.040 218.000 C 206.502 218.000,207.492 217.388,208.240 216.640 C 208.988 215.892,209.600 214.902,209.600 214.440 C 209.600 213.811,209.851 213.600,210.600 213.600 L 211.600 213.600 211.600 205.408 L 211.600 197.215 210.600 196.964 C 209.923 196.794,209.600 196.436,209.599 195.857 C 209.598 195.351,209.107 194.672,208.400 194.200 C 207.741 193.760,207.202 193.265,207.201 193.100 C 207.201 192.935,206.750 192.800,206.200 192.800 C 205.400 192.800,205.200 192.600,205.200 191.800 L 205.200 190.800 197.067 190.800 C 192.593 190.800,188.813 190.920,188.667 191.067 M205.467 231.067 C 205.320 231.213,205.200 231.653,205.200 232.044 C 205.200 232.593,204.816 232.782,203.500 232.877 C 202.198 232.971,201.709 233.207,201.412 233.884 C 201.199 234.370,200.659 234.820,200.212 234.884 C 199.534 234.981,199.380 235.342,199.280 237.075 C 199.169 238.997,199.080 239.160,198.080 239.275 L 197.000 239.400 197.000 247.400 L 197.000 255.400 198.080 255.525 C 199.080 255.640,199.169 255.803,199.280 257.725 C 199.383 259.510,199.522 259.817,200.275 259.925 C 200.821 260.002,201.198 260.379,201.275 260.925 C 201.380 261.662,201.696 261.819,203.275 261.921 C 204.979 262.031,205.162 262.141,205.275 263.121 L 205.400 264.200 213.600 264.200 L 221.800 264.200 221.927 263.100 C 222.045 262.069,222.175 262.000,223.983 262.000 C 225.651 262.000,225.944 261.878,226.140 261.100 C 226.265 260.605,226.779 260.020,227.283 259.800 C 228.023 259.477,228.223 259.034,228.322 257.500 C 228.432 255.785,228.539 255.600,229.422 255.600 L 230.400 255.600 230.400 247.400 L 230.400 239.200 229.422 239.200 C 228.540 239.200,228.432 239.015,228.322 237.310 C 228.222 235.765,228.020 235.330,227.217 234.931 C 226.676 234.663,226.097 234.074,225.930 233.622 C 225.684 232.958,225.276 232.800,223.813 232.800 C 222.146 232.800,222.000 232.720,222.000 231.800 L 222.000 230.800 213.867 230.800 C 209.393 230.800,205.613 230.920,205.467 231.067 M164.254 261.113 C 164.143 261.292,164.196 261.670,164.371 261.953 C 164.619 262.354,164.741 262.331,164.927 261.846 C 165.179 261.190,164.598 260.557,164.254 261.113 M186.508 261.620 C 186.448 262.419,186.979 262.681,187.372 262.045 C 187.493 261.850,187.361 261.499,187.079 261.266 C 186.694 260.946,186.553 261.035,186.508 261.620 "
+                  stroke="none"
+                  fill="#cc0434"
+                  fill-rule="evenodd"
+                ></path>
+                <path
+                  id="path3"
+                  d="M123.477 131.856 C 123.325 132.009,123.200 132.553,123.200 133.067 C 123.200 133.798,122.981 134.000,122.190 134.000 L 121.179 134.000 121.290 140.100 C 121.397 146.031,121.424 146.203,122.275 146.325 C 122.821 146.402,123.198 146.779,123.275 147.325 C 123.397 148.176,123.569 148.203,129.500 148.310 L 135.600 148.421 135.600 147.437 C 135.600 146.661,135.832 146.426,136.700 146.327 L 137.800 146.200 137.800 140.200 L 137.800 134.200 136.724 134.076 C 135.893 133.980,135.620 133.707,135.524 132.876 L 135.400 131.800 129.577 131.690 C 126.374 131.629,123.629 131.704,123.477 131.856 M108.572 174.968 C 108.421 175.571,107.990 176.002,107.462 176.077 C 106.075 176.275,106.112 190.131,107.500 190.328 C 107.995 190.398,108.400 190.700,108.400 190.999 C 108.400 192.534,109.470 192.800,115.655 192.800 C 121.866 192.800,123.200 192.520,123.200 191.214 C 123.200 190.732,123.528 190.409,124.100 190.328 C 125.499 190.129,125.523 176.274,124.125 176.075 C 123.579 175.998,123.202 175.621,123.125 175.075 C 123.002 174.217,122.860 174.198,115.910 174.091 L 108.820 173.982 108.572 174.968 M102.400 216.973 C 102.400 217.738,102.164 217.974,101.300 218.073 C 100.523 218.163,100.163 218.457,100.075 219.075 C 99.998 219.621,99.621 219.998,99.075 220.075 C 98.212 220.198,98.200 220.298,98.200 227.400 C 98.200 234.502,98.212 234.602,99.075 234.725 C 99.621 234.802,99.998 235.179,100.075 235.725 C 100.163 236.343,100.523 236.637,101.300 236.727 C 102.164 236.826,102.400 237.062,102.400 237.827 L 102.400 238.800 109.600 238.800 L 116.800 238.800 116.800 237.800 C 116.800 237.000,117.000 236.800,117.800 236.800 C 118.593 236.800,118.800 236.598,118.800 235.827 C 118.800 235.062,119.036 234.826,119.900 234.727 L 121.000 234.600 121.110 227.853 C 121.224 220.835,121.058 220.000,119.551 220.000 C 119.017 220.000,118.800 219.712,118.800 219.000 C 118.800 218.200,118.600 218.000,117.800 218.000 C 117.000 218.000,116.800 217.800,116.800 217.000 L 116.800 216.000 109.600 216.000 L 102.400 216.000 102.400 216.973 M113.280 258.480 C 113.016 258.744,112.800 259.181,112.800 259.452 C 112.800 259.723,112.395 260.002,111.900 260.072 C 111.256 260.164,110.964 260.512,110.873 261.300 C 110.783 262.082,110.521 262.400,109.964 262.400 C 108.707 262.400,108.400 264.012,108.400 270.600 C 108.400 277.188,108.707 278.800,109.964 278.800 C 110.521 278.800,110.783 279.118,110.873 279.900 C 110.959 280.647,111.260 281.037,111.812 281.116 C 112.259 281.180,112.799 281.630,113.012 282.116 C 113.390 282.978,113.590 283.003,121.107 283.109 L 128.813 283.217 129.185 282.239 C 129.402 281.670,129.942 281.207,130.478 281.131 C 131.140 281.037,131.435 280.696,131.524 279.925 C 131.611 279.172,131.911 278.812,132.524 278.725 C 133.905 278.528,133.905 262.672,132.524 262.475 C 131.911 262.388,131.611 262.028,131.524 261.275 C 131.435 260.504,131.140 260.163,130.478 260.069 C 129.945 259.994,129.402 259.531,129.188 258.969 L 128.820 258.000 121.290 258.000 C 115.967 258.000,113.619 258.141,113.280 258.480 M140.276 291.858 C 140.124 292.009,140.000 292.543,140.000 293.045 C 140.000 293.848,139.775 293.971,138.100 294.078 L 136.200 294.200 136.078 296.100 C 135.968 297.815,135.861 298.000,134.978 298.000 L 134.000 298.000 134.000 305.400 L 134.000 312.800 134.978 312.800 C 135.861 312.800,135.968 312.985,136.078 314.700 L 136.200 316.600 138.100 316.722 C 139.815 316.832,140.000 316.939,140.000 317.822 L 140.000 318.800 147.400 318.800 L 154.800 318.800 154.800 317.822 C 154.800 316.939,154.985 316.832,156.700 316.722 L 158.600 316.600 158.721 314.725 C 158.831 313.021,158.941 312.838,159.921 312.725 L 161.000 312.600 161.000 305.400 L 161.000 298.200 159.921 298.075 C 158.941 297.962,158.831 297.779,158.721 296.075 L 158.600 294.200 156.725 294.079 C 155.021 293.969,154.838 293.859,154.725 292.879 L 154.600 291.800 147.576 291.691 C 143.712 291.631,140.427 291.706,140.276 291.858 M230.800 313.778 C 230.800 314.661,230.615 314.768,228.900 314.878 C 227.396 314.975,226.920 315.182,226.616 315.874 C 226.405 316.355,225.820 316.936,225.316 317.165 C 224.558 317.511,224.400 317.894,224.400 319.391 C 224.400 321.058,224.321 321.200,223.392 321.200 L 222.384 321.200 222.492 329.500 C 222.599 337.706,222.610 337.801,223.479 337.925 C 224.221 338.031,224.377 338.344,224.479 339.925 C 224.582 341.521,224.734 341.819,225.500 341.928 C 226.083 342.011,226.400 342.331,226.400 342.838 C 226.400 343.715,227.803 344.400,229.598 344.400 C 230.482 344.400,230.774 344.606,230.872 345.300 C 231.000 346.199,231.008 346.200,239.000 346.200 C 246.992 346.200,247.000 346.199,247.128 345.300 C 247.226 344.606,247.518 344.400,248.402 344.400 C 250.197 344.400,251.600 343.715,251.600 342.838 C 251.600 342.331,251.917 342.011,252.500 341.928 C 253.266 341.819,253.418 341.521,253.521 339.925 C 253.623 338.344,253.779 338.031,254.521 337.925 C 255.390 337.801,255.401 337.706,255.508 329.500 L 255.616 321.200 254.608 321.200 C 253.668 321.200,253.600 321.068,253.600 319.243 C 253.600 317.548,253.480 317.256,252.700 317.060 C 252.205 316.935,251.620 316.421,251.400 315.917 C 251.077 315.177,250.634 314.977,249.100 314.878 C 247.385 314.768,247.200 314.661,247.200 313.778 L 247.200 312.800 239.000 312.800 L 230.800 312.800 230.800 313.778 M178.500 317.065 C 178.225 317.176,178.000 317.702,178.000 318.233 C 178.000 319.002,177.792 319.200,176.982 319.200 C 175.671 319.200,174.000 320.871,174.000 322.182 C 174.000 322.896,173.785 323.200,173.280 323.200 C 171.745 323.200,171.600 323.910,171.600 331.447 C 171.600 339.546,171.677 340.000,173.043 340.000 C 173.743 340.000,174.000 340.221,174.000 340.826 C 174.000 342.613,175.525 344.365,177.100 344.387 C 177.712 344.396,178.000 344.642,178.000 345.157 C 178.000 346.300,178.674 346.400,186.400 346.400 C 194.126 346.400,194.800 346.300,194.800 345.157 C 194.800 344.660,195.082 344.400,195.623 344.400 C 196.691 344.400,198.800 342.084,198.800 340.911 C 198.800 340.217,199.028 340.000,199.757 340.000 C 201.123 340.000,201.200 339.546,201.200 331.447 C 201.200 323.910,201.055 323.200,199.520 323.200 C 199.045 323.200,198.800 322.901,198.800 322.322 C 198.800 321.149,196.881 319.200,195.725 319.200 C 195.016 319.200,194.800 318.976,194.800 318.243 C 194.800 317.716,194.515 317.176,194.167 317.043 C 193.414 316.754,179.221 316.774,178.500 317.065 "
+                  stroke="none"
+                  fill="#0464fc"
+                  fill-rule="evenodd"
+                ></path>
+              </g>
+            </svg>
+            <div class="flex-grow"></div>
+            <a
+              class="card-nav"
+              href="https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#how-to"
+              target="_blank"
+            >
+              <svg
+                xmlns="http://www.w3.org/2000/svg"
+                class="h-6 w-6"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M17 8l4 4m0 0l-4 4m4-4H3"
+                />
+              </svg>
+              Extending the benchmark
+            </a>
+          </div>
+        </div>
+      </section>
+      <section>
+        <div class="two-col">
+          <div class="flat-card">
+            <h3>📄 Paper</h3>
+            <div>
+              A preprint of our most recent paper is available on
+              <a href="https://arxiv.org/abs/2207.12560" target="_blank"
+                >ArXiv</a
+              >. It includes an in-depth discussion of the different design
+              decisions and its limiations, as well as a multi-faceted analysis
+              of results from large scale comparison across 9 frameworks on more
+              than 100 tasks.
+              <!-- https://arxiv.org/abs/1907.00909 -->
+            </div>
+          </div>
+          <div class="flat-card">
+            <h3>🧑‍💻 Code</h3>
+            <div>
+              The entire benchmark tool is open source and developed on
+              <a
+                href="https://github.com/openml/automlbenchmark"
+                target="_blank"
+                >Github</a
+              >. The Github discussion board and issue trackers are the main way
+              for us to interact with the community.
+            </div>
+          </div>
+        </div>
+        <!-- <h2>Paper <img src=/></h2> -->
+      </section>
+      <section>
+        <h2>AutoML Frameworks</h2>
+        <div>
+          <div class="grey">
+            Many AutoML frameworks are already integrated with the AutoML
+            benchmark tool and
+            <a href="#ADDFRAMEWORK">adding more is easy</a>. We have more
+            information about the different frameworks on our
+            <a href="#FRAMEWORKS">framework overview page</a>. The icons below
+            link directly to their respective Github repositories.
+          </div>
+          <div class="icon-flex">
+            <a
+              href="https://github.com/openml-labs/gama"
+              target="_blank"
+              class="framework-logo"
+              ><img src="img/logos/GAMA-icon.png" title="GAMA"
+            /></a>
+            <a href="https://github.com/EpistasisLab/tpot" target="_blank"
+              ><img src="img/logos/tpot.jpeg" title="TPOT"
+            /></a>
+            <a href="https://github.com/awslabs/autogluon" target="_blank"
+              ><img src="img/logos/autogluon.png" title="AutoGluon"
+            /></a>
+            <a
+              href="https://github.com/automl/auto-sklearn"
+              target="_blank"
+              class="framework-logo"
+              ><img src="img/logos/auto-sklearn.png" title="auto-sklearn"
+            /></a>
+            <a href="https://github.com/mljar/mljar-supervised" target="_blank"
+              ><img src="img/logos/mljar.png" title="mljar-supervised"
+            /></a>
+            <a href="https://github.com/h2oai/h2o-3" target="_blank"
+              ><img src="img/logos/h2o-automl-logo.jpeg" title="H2O AutoML"
+            /></a>
+            <a
+              href="https://github.com/sb-ai-lab/LightAutoML"
+              target="_blank"
+              class="framework-logo"
+              ><img
+                src="img/logos/LightAutoML_logo_small.png"
+                title="Light AutoML"
+            /></a>
+            <a href="https://github.com/microsoft/FLAML" target="_blank"
+              ><img src="img/logos/flaml.svg" title="FLAML"
+            /></a>
+          </div>
+          <div></div>
+        </div>
+      </section>
+      <section>
+        <h2>Community</h2>
+        <div class="grey">
+          We welcome any contributions to the AutoML benchmark. Our goal is to
+          provide the best benchmark tools for AutoML research and we can't do
+          that on our own. Contributions are appreciated in many forms,
+          including feedback on the benchmark design, feature requests, bug
+          reports, code and documentation contributions, and more. Why not stop
+          by on our
+          <a
+            href="https://github.com/openml/automlbenchmark/discussions/216"
+            target="_blank"
+            >welcome board</a
+          >
+          and let us know what got you interested in the benchmark?
+        </div>
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/papers.html b/docs/papers.html
new file mode 100644
index 000000000..3d25ceaaa
--- /dev/null
+++ b/docs/papers.html
@@ -0,0 +1,308 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Papers</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <h1 class="page-title">Papers</h1>
+      <div class="paper">
+        <h2 class="paper-title">AMLB: an AutoML Benchmark</h2>
+        <div class="paper-authors">
+          Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell,
+          Sébastien Poirier, Janek Thomas, Bernd Bischl and Joaquin Vanschoren
+        </div>
+
+        <!-- <div class="paper-year">2019</div>
+      <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+      <div class="paper-venue">cite</div> -->
+
+        <div class="paper-abstract">
+          Comparing different AutoML frameworks is notoriously challenging and
+          often done incorrectly. We introduce an open and extensible benchmark
+          that follows best practices and avoids common mistakes when comparing
+          AutoML frameworks. We conduct a thorough comparison of 9 well-known
+          AutoML frameworks across 71 classification and 33 regression tasks.
+          The differences between the AutoML frameworks are explored with a
+          multi-faceted analysis, evaluating model accuracy, its trade-offs with
+          inference time, and framework failures. We also use Bradley-Terry
+          trees to discover subsets of tasks where the relative AutoML framework
+          rankings differ. The benchmark comes with an open-source tool that
+          integrates with many AutoML frameworks and automates the empirical
+          evaluation process end-to-end: from framework installation and
+          resource allocation to in-depth evaluation. The benchmark uses public
+          data sets, can be easily extended with other AutoML frameworks and
+          tasks, and has a website with up-to-date results.
+        </div>
+        <div class="paper-links">
+          <div class="hover-expand">
+            <strong>2022</strong>
+            <div>Under Review</div>
+          </div>
+          <a class="flat-button">
+            <label for="cite-2022">CITE</label>
+          </a>
+          <a href="https://arxiv.org/pdf/2207.12560.pdf" target="_blank">PDF</a>
+          <a href="https://arxiv.org/abs/2207.12560" target="_blank">ArXiv</a>
+        </div>
+        <div>
+          <input
+            type="checkbox"
+            id="cite-2022"
+            class="hidden-content-checkbox"
+          />
+          <div class="hidden-content bibtex">
+            @misc{https://doi.org/10.48550/arxiv.2207.12560, <br />
+            &emsp; doi = {10.48550/ARXIV.2207.12560}, <br />
+            &emsp; url = {https://arxiv.org/abs/2207.12560}, <br />
+            &emsp; author = {Gijsbers, Pieter and Bueno, Marcos L. P. and Coors,
+            Stefan and LeDell, Erin and Poirier, S\'{e}bastien and Thomas, Janek
+            and Bischl, Bernd and Vanschoren, Joaquin}, <br />
+            &emsp; keywords = {Machine Learning (cs.LG), Machine Learning
+            (stat.ML), FOS: Computer and information sciences, FOS: Computer and
+            information sciences}, <br />
+            &emsp; title = {AMLB: an AutoML Benchmark}, <br />
+            &emsp; publisher = {arXiv}, <br />
+            &emsp; year = {2022}, <br />
+            &emsp; copyright = {Creative Commons Attribution 4.0 International}
+            <br />
+            }
+          </div>
+        </div>
+      </div>
+      <div class="paper">
+        <h2 class="paper-title">An Open Source AutoML Benchmark</h2>
+        <div class="paper-authors">
+          Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd
+          Bischl, Joaquin Vanschoren
+        </div>
+
+        <!-- <div class="paper-year">2019</div>
+      <div class="paper-venue">ICML 2019 AutoML Workshop</div>
+      <div class="paper-venue">cite</div> -->
+
+        <div class="paper-abstract">
+          In recent years, an active field of research has developed around
+          automated machine learning (AutoML). Unfortunately, comparing
+          different AutoML systems is hard and often done incorrectly. We
+          introduce an open, ongoing, and extensible benchmark framework which
+          follows best practices and avoids common mistakes. The framework is
+          open-source, uses public datasets and has a website with up-to-date
+          results. We use the framework to conduct a thorough comparison of 4
+          AutoML systems across 39 datasets and analyze the results.
+        </div>
+        <div class="paper-links">
+          <div class="hover-expand">
+            <strong>2019</strong>
+            <div>ICML 2019 AutoML Workshop</div>
+          </div>
+          <a class="flat-button">
+            <label for="cite-2019">CITE</label>
+          </a>
+          <a
+            href="https://www.automl.org/wp-content/uploads/2019/06/automlws2019_Paper45.pdf"
+            target="_blank"
+            >PDF</a
+          >
+          <a href="https://arxiv.org/abs/1907.00909" target="_blank">ArXiv</a>
+        </div>
+        <div>
+          <input
+            type="checkbox"
+            id="cite-2019"
+            class="hidden-content-checkbox"
+          />
+          <div class="hidden-content bibtex">
+            @article{amlb2019, <br />
+            &emsp; title={An Open Source AutoML Benchmark},<br />
+            &emsp; author={Gijsbers, P. and LeDell, E. and Poirier, S. and
+            Thomas, J. and Bischl, B. and Vanschoren, J.}, <br />
+            &emsp; journal={arXiv preprint arXiv:1907.00909 [cs.LG]}, <br />
+            &emsp; url={https://arxiv.org/abs/1907.00909}, <br />
+            &emsp; note={Accepted at AutoML Workshop at ICML 2019}, <br />
+            &emsp; year={2019} <br />}
+          </div>
+        </div>
+      </div>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/results.html b/docs/results.html
new file mode 100644
index 000000000..504ee0f07
--- /dev/null
+++ b/docs/results.html
@@ -0,0 +1,281 @@
+<html>
+  <head>
+    <meta charset="utf-8" />
+    <title>AMLB: Results</title>
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <link rel="preconnect" href="https://fonts.googleapis.com" />
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+    <link
+      href="https://fonts.googleapis.com/css2?family=Inconsolata&family=Lato:ital,wght@0,400;0,700;1,400&display=swap"
+      rel="stylesheet"
+    />
+    <link href="style.css" rel="stylesheet" />
+  </head>
+  <body>
+    <nav class="navigation-bar desktop">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">Results</a>
+      <a href="frameworks.html" class="nav-link">Frameworks</a>
+      <a href="papers.html" class="nav-link">papers</a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        >user guide
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+        >Discussions
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon-with-text"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="100%" />
+      </a>
+    </nav>
+
+    <nav class="navigation-bar mobile nav-mobile">
+      <a href="index.html">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M10.707 2.293a1 1 0 00-1.414 0l-7 7a1 1 0 001.414 1.414L4 10.414V17a1 1 0 001 1h2a1 1 0 001-1v-2a1 1 0 011-1h2a1 1 0 011 1v2a1 1 0 001 1h2a1 1 0 001-1v-6.586l.293.293a1 1 0 001.414-1.414l-7-7z"
+          />
+        </svg>
+      </a>
+      <a href="results.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M5 3a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2V5a2 2 0 00-2-2H5zm9 4a1 1 0 10-2 0v6a1 1 0 102 0V7zm-3 2a1 1 0 10-2 0v4a1 1 0 102 0V9zm-3 3a1 1 0 10-2 0v1a1 1 0 102 0v-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a href="frameworks.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M5 4a1 1 0 00-2 0v7.268a2 2 0 000 3.464V16a1 1 0 102 0v-1.268a2 2 0 000-3.464V4zM11 4a1 1 0 10-2 0v1.268a2 2 0 000 3.464V16a1 1 0 102 0V8.732a2 2 0 000-3.464V4zM16 3a1 1 0 011 1v7.268a2 2 0 010 3.464V16a1 1 0 11-2 0v-1.268a2 2 0 010-3.464V4a1 1 0 011-1z"
+          />
+        </svg>
+      </a>
+      <a href="papers.html" class="nav-link">
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M9 4.804A7.968 7.968 0 005.5 4c-1.255 0-2.443.29-3.5.804v10A7.969 7.969 0 015.5 14c1.669 0 3.218.51 4.5 1.385A7.962 7.962 0 0114.5 14c1.255 0 2.443.29 3.5.804v-10A7.968 7.968 0 0014.5 4c-1.255 0-2.443.29-3.5.804V12a1 1 0 11-2 0V4.804z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark#readme"
+        target="_blank"
+        class="nav-link"
+        ><svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            fill-rule="evenodd"
+            d="M10.496 2.132a1 1 0 00-.992 0l-7 4A1 1 0 003 8v7a1 1 0 100 2h14a1 1 0 100-2V8a1 1 0 00.496-1.868l-7-4zM6 9a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1zm3 1a1 1 0 012 0v3a1 1 0 11-2 0v-3zm5-1a1 1 0 00-1 1v3a1 1 0 102 0v-3a1 1 0 00-1-1z"
+            clip-rule="evenodd"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/discussions/216"
+        target="_blank"
+        class="nav-link"
+      >
+        <svg
+          xmlns="http://www.w3.org/2000/svg"
+          class="nav-icon"
+          viewBox="0 0 20 20"
+          fill="currentColor"
+        >
+          <path
+            d="M2 5a2 2 0 012-2h7a2 2 0 012 2v4a2 2 0 01-2 2H9l-3 3v-3H4a2 2 0 01-2-2V5z"
+          />
+          <path
+            d="M15 7v2a4 4 0 01-4 4H9.828l-1.766 1.767c.28.149.599.233.938.233h2l3 3v-3h2a2 2 0 002-2V9a2 2 0 00-2-2h-1z"
+          />
+        </svg>
+      </a>
+      <a
+        href="https://github.com/openml/automlbenchmark/"
+        target="_blank"
+        class="nav-icon"
+      >
+        <img src="img/logos/GitHub-Mark-Light-64px.png" height="22px" />
+      </a>
+    </nav>
+    <!-- <div class="construction-banner">
+      🚧 This website is under construction 🚧
+    </div> -->
+    <div class="page-content">
+      <section>
+        <h1 class="page-title">Results</h1>
+        <div class="accordion acard">
+          <div>
+            ⚠️ <a href="papers.html">Our paper</a> outlines
+            <strong>important limitations</strong> for the interpretation of
+            results. These limitations include:
+          </div>
+          <input
+            type="checkbox"
+            id="more-limitations"
+            class="accordion-input"
+          />
+          <div class="accordion-content">
+            <ul class="limitations-list">
+              <li>
+                We use AutoML framework versions from
+                <strong>September 2021</strong>, many frameworks have since seen
+                major updates.
+              </li>
+              <li>
+                We use the "benchmark" modes of the frameworks, which generally
+                <em>only</em> optimize for performance. Most AutoML frameworks
+                have multiple modes to support different use cases.
+              </li>
+              <li>
+                Results can not be used to make conclusions about which
+                algorithm is best, as all frameworks differ in multiple ways.
+              </li>
+              <li>
+                Performance statistics are often independent from many
+                qualitative differences, such as ease of use or
+                interpretability.
+              </li>
+            </ul>
+            Please read Section 5.3 in our paper for a more elaborate discussion
+            of these and other limitations.
+          </div>
+          <label for="more-limitations">
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-down accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M19 9l-7 7-7-7"
+              />
+            </svg>
+            <svg
+              xmlns="http://www.w3.org/2000/svg"
+              class="accordion-chevron-up accordion-icon"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M5 15l7-7 7 7"
+              />
+            </svg>
+          </label>
+        </div>
+      </section>
+      <section class="results">
+        <div>
+          All results are available as
+          <a href="https://test.openml.org/amlb/" target="_blank">raw files</a>
+          📂, but we open source the tools we used for generating the figures in
+          our paper. The best way to explore the results is through our
+          interactive
+          <a
+            href="https://compstat-lmu.shinyapps.io/AutoML-Benchmark-Analysis/"
+            target="_blank"
+            >Shiny app</a
+          >. It loads the latest results by default. It is also possible to use
+          our
+          <a
+            href="https://github.com/openml/automlbenchmark/tree/master/reports"
+            target="_blank"
+            >notebooks</a
+          >
+          which contain additional visualizations.
+        </div>
+        <img src="img/shiny.png" height="320px" class="shiny" />
+      </section>
+    </div>
+  </body>
+  <footer>
+    © 2022 Copyright, the AutoML Benchmark team.<br />
+    Icons of AutoML frameworks and OpenML by their respective owners. <br />
+    Spotted an issue?
+    <a href="https://github.com/openml/automlbenchmark" target="_blank"
+      >Tell us on Github or open a pull request</a
+    >!
+  </footer>
+</html>
diff --git a/docs/style.css b/docs/style.css
new file mode 100644
index 000000000..73340f289
--- /dev/null
+++ b/docs/style.css
@@ -0,0 +1,527 @@
+* {
+  margin: 0;
+  padding: 0;
+  box-sizing: border-box;
+  font-size: 18px;
+}
+.grey {
+  color: #495057;
+}
+
+/* main: #1971c2 */
+
+body {
+  font-family: "lato", sans-serif;
+  color: #343a40;
+}
+
+section {
+  margin-bottom: 64px;
+}
+
+a {
+  color: 1971c2;
+}
+
+.page-content {
+  padding: 0 clamp(15%, 100%, calc((100vw - 1000px) / 2)); /* should maybe set width in px instead */
+}
+
+h1 {
+  font-size: 72px;
+  padding-bottom: 24px;
+}
+
+h2 {
+  font-size: 48px;
+  padding-bottom: 24px;
+}
+
+h3 {
+  font-size: 32px;
+  padding-bottom: 16px;
+}
+
+footer {
+  color: #e7f5ff;
+  font-size: 14px;
+  background-color: #1971c2;
+  text-align: center;
+  padding: 8px;
+}
+
+footer > a {
+  color: #e7f5ff;
+  font-size: 14px;
+}
+
+.terminal {
+  background-color: #343a40;
+  color: #f8f9fa;
+  font-family: "Inconsolata", sans-serif;
+  padding: 8px;
+  margin: 16px 5%;
+}
+
+/* NAVBAR */
+.construction-banner {
+  background-color: #1c7ed6;
+  color: white;
+  text-align: center;
+}
+
+.navigation-bar {
+  background-color: #1971c2;
+  color: white;
+  text-transform: uppercase;
+
+  display: flex;
+  gap: 24px;
+  justify-content: flex-end;
+  align-items: center;
+  padding: 16px;
+  height: 60px;
+}
+
+nav a:first-child {
+  flex-grow: 1;
+}
+
+.nav-icon {
+  height: 24px;
+  fill: white;
+}
+
+.nav-icon-with-text {
+  height: 18px;
+}
+
+.nav-icon:hover,
+.nav-icon:active {
+  height: 28px;
+}
+
+.nav-link:link,
+.nav-link:visited {
+  text-decoration: none;
+  color: white;
+}
+.nav-link:hover,
+.nav-link:active {
+  color: white;
+  font-weight: bold;
+}
+
+/* TITLE */
+
+.title,
+.subtitle {
+  color: #343a40;
+  text-align: center;
+}
+
+.title {
+  font-size: 74px;
+  padding-top: 48px;
+  font-weight: bold;
+}
+
+.subtitle {
+  font-size: 62px;
+  font-weight: bold;
+  padding-bottom: 24px;
+}
+
+.summary {
+  font-size: 24px;
+  color: #495057;
+  padding: 0 max((100% - 900px) / 2, 0px); /* does not seem to work with just the math expression */
+}
+
+/* CARDS */
+
+.three-cols {
+  display: grid;
+  grid-template-columns: repeat(3, 1fr);
+  column-gap: 5%;
+}
+
+.card {
+  box-shadow: 0 5px 10px rgba(0, 0, 0, 0.1);
+  border-radius: 5px;
+  padding: 16px;
+
+  display: flex;
+  flex-direction: column;
+}
+
+.card img {
+  padding: 16px 2%;
+}
+
+.card > svg {
+  display: block;
+  margin: 16px auto;
+}
+
+.flex-grow {
+  flex-grow: 1;
+}
+
+.card > .card-nav {
+  color: #1971c2;
+  text-decoration: none;
+
+  display: flex;
+  align-items: center;
+  justify-content: flex-start;
+  gap: 8px;
+}
+
+.card > .card-nav > svg {
+  height: 24px;
+  fill: #1971c2;
+}
+
+h3 {
+  font-size: 24px;
+  padding-bottom: 10px;
+}
+
+/* Frameworks */
+/* .icon-grid {
+  padding: 24px;
+  display: grid;
+  grid-auto-rows: 60px;
+} */
+.icon-flex {
+  margin-top: 24px;
+  display: flex;
+  flex-wrap: wrap;
+  justify-content: center;
+  align-items: center;
+  gap: 40px;
+}
+
+.icon-flex a img {
+  height: 90px;
+}
+/* Paper */
+
+.two-col {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  column-gap: 2%;
+
+  background-color: #1971c2;
+  color: #e7f5ff;
+  border-radius: 5px;
+}
+
+.two-col a {
+  color: white;
+}
+
+.two-col h3 {
+  color: white;
+}
+
+.flat-card {
+  padding: 16px;
+}
+
+/* FRAMEWORKS PAGE */
+.page-title {
+  margin-top: 72px;
+}
+
+.framework-card-list {
+  background-color: #e7f5ff;
+  padding: 32px 5%;
+  border-radius: 10px;
+}
+
+.framework-card-list > .accordion {
+  margin-bottom: 32px;
+}
+/* 
+.framework-card {
+  background-color: white;
+  margin-bottom: 64;
+  padding: 32;
+  border-radius: 10px;
+} */
+.framework-card:last-of-type {
+  margin-bottom: 0;
+}
+
+.framework-header {
+  display: flex;
+  align-items: center;
+  gap: 16px;
+  flex-wrap: wrap;
+}
+
+.framework-header > h3 {
+  padding-bottom: 0px;
+  font-size: 32px;
+}
+
+.framework-links {
+  display: flex;
+  gap: 16px;
+  justify-content: flex-end;
+
+  flex-grow: 1;
+}
+
+.framework-links > a {
+  text-decoration: none;
+  font-size: 24px;
+}
+
+/* RESULTS */
+
+.limitations-list {
+  padding: 0px 32px 16px 32px;
+}
+
+.limitations-list li {
+  padding: 8px 0px;
+}
+
+.results {
+  display: flex;
+  gap: 40px;
+}
+
+/* Accordions */
+.accordion {
+}
+
+.acard {
+  box-shadow: 0 5px 10px rgba(0, 0, 0, 0.2);
+  border-radius: 10px;
+  background-color: white;
+}
+
+.accordion > div {
+  padding: 8px 32px;
+}
+
+.accordion > div:first-child {
+  padding-top: 16px;
+}
+
+.accordion > label {
+  width: 100%;
+  display: flex;
+  justify-content: center;
+}
+
+.accordion-input {
+  display: none;
+}
+
+.accordion-icon {
+  height: 24px;
+}
+
+.accordion-input:checked ~ .accordion-content {
+  display: block;
+}
+
+.accordion-input:checked ~ label .accordion-chevron-down {
+  display: none;
+}
+
+.accordion-input:checked ~ label .accordion-chevron-up {
+  display: block;
+}
+
+.accordion-input ~ .accordion-content {
+  display: none;
+}
+
+.accordion-input ~ label .accordion-chevron-up {
+  display: none;
+}
+
+.accordion-input ~ label .accordion-chevron-down {
+  display: block;
+}
+
+.accordion > label {
+  background: linear-gradient(0deg, rgba(0, 0, 0, 0.02), white);
+  border-radius: 10px;
+  padding-bottom: 8px;
+}
+
+.accordion > label:hover {
+  cursor: pointer;
+}
+
+/* Papers */
+.paper {
+  margin-bottom: 48px;
+}
+
+.paper-title {
+  padding-bottom: 0px;
+  margin-bottom: 0px;
+}
+
+.paper-links {
+  display: flex;
+  align-items: center;
+  gap: 12px;
+
+  margin-bottom: 16px;
+}
+
+.paper-links > a {
+  color: white;
+  background-color: #1c7ed6;
+  padding: 4px 12px;
+  border-radius: 4px;
+
+  text-decoration: none;
+  text-transform: uppercase;
+}
+
+.paper-year {
+  color: #343a40;
+  display: inline;
+  font-weight: bold;
+}
+
+.paper-venue {
+  color: #495057;
+  display: inline;
+}
+
+.paper-authors {
+  color: #4263eb;
+  padding-bottom: 8px;
+}
+
+.paper-abstract {
+  color: #495057;
+  padding-bottom: 16px;
+}
+
+.hover-expand {
+  display: flex;
+  gap: 20px;
+
+  color: #343a40;
+  background-color: white;
+  border: 2px solid #495057;
+  border-radius: 4px;
+  padding: 2px 8px;
+}
+
+.hover-expand > div {
+  display: none;
+}
+
+.hover-expand:hover > div {
+  display: block;
+}
+
+.hidden-content {
+  display: none;
+}
+
+.hidden-content-checkbox {
+  display: none;
+}
+
+.hidden-content-checkbox:checked ~ .hidden-content {
+  display: block;
+}
+
+.bibtex {
+  font-family: "Inconsolata", sans-serif;
+  color: #343a40;
+  background-color: #e7f5ff;
+  border-radius: 10px;
+  padding: 16px;
+}
+
+.flat-button {
+  cursor: pointer;
+}
+
+.flat-button > label {
+  cursor: pointer;
+}
+
+.mobile {
+  display: none;
+}
+
+.nav-mobile {
+  justify-content: space-between;
+  display: none;
+}
+
+@media (max-width: 900px) {
+  .desktop {
+    display: none;
+  }
+
+  .mobile {
+    display: block;
+    padding: 25px;
+  }
+
+  .shiny {
+    height: auto;
+    width: 100%;
+  }
+
+  .results {
+    flex-wrap: wrap;
+  }
+
+  .framework-header > h3 {
+    order: -1;
+    flex-basis: 100%;
+  }
+
+  h1 {
+    font-size: 48px;
+    padding-bottom: 18px;
+  }
+
+  .page-title {
+    margin-top: 32px;
+  }
+
+  .page-content {
+    padding: 0 32px;
+  }
+
+  .three-cols {
+    grid-template-columns: 1fr;
+    row-gap: 32px;
+  }
+
+  .two-col {
+    grid-template-columns: 1fr;
+    row-gap: 32px;
+  }
+
+  .nav-mobile {
+    display: flex;
+  }
+}
+
+.mobile * {
+  color: white;
+}
+
+.mobile > .summary > a {
+  font-size: 24px;
+}
diff --git a/frameworks/AutoGluon/README.md b/frameworks/AutoGluon/README.md
new file mode 100644
index 000000000..51286533e
--- /dev/null
+++ b/frameworks/AutoGluon/README.md
@@ -0,0 +1,16 @@
+# AutoGluon
+
+To run v0.5.2: ```python3 ../automlbenchmark/runbenchmark.py autogluon ...```
+
+To run mainline: ```python3 ../automlbenchmark/runbenchmark.py autogluonts:latest ...```
+
+
+# AutoGluonTS
+
+AutoGluonTS stands for autogluon.timeseries. This framework handles time series problems.
+
+## Run Steps
+
+To run v0.5.2: ```python3 ../automlbenchmark/runbenchmark.py autogluonts timeseries ...```
+
+To run mainline: ```python3 ../automlbenchmark/runbenchmark.py autogluonts:latest timeseries ...```
diff --git a/frameworks/AutoGluon/__init__.py b/frameworks/AutoGluon/__init__.py
index be2c15147..9d3d980a3 100644
--- a/frameworks/AutoGluon/__init__.py
+++ b/frameworks/AutoGluon/__init__.py
@@ -1,15 +1,23 @@
-from amlb.benchmark import TaskConfig
-from amlb.data import Dataset
+
 from amlb.utils import call_script_in_same_dir
+from amlb.benchmark import TaskConfig
+from amlb.data import Dataset, DatasetType
+from copy import deepcopy
 
 
 def setup(*args, **kwargs):
     call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
 
-
 def run(dataset: Dataset, config: TaskConfig):
-    from frameworks.shared.caller import run_in_venv
 
+    if dataset.type is not DatasetType.timeseries:
+        return run_autogluon_tabular(dataset, config)
+
+    else:
+        return run_autogluon_timeseries(dataset, config)
+
+def run_autogluon_tabular(dataset: Dataset, config: TaskConfig):
+    from frameworks.shared.caller import run_in_venv
     data = dict(
         train=dict(path=dataset.train.data_path('parquet')),
         test=dict(path=dataset.test.data_path('parquet')),
@@ -17,9 +25,38 @@ def run(dataset: Dataset, config: TaskConfig):
             name=dataset.target.name,
             classes=dataset.target.values
         ),
-        problem_type=dataset.type.name  # AutoGluon problem_type is using same names as amlb.data.DatasetType
+        problem_type=dataset.type.name,  # AutoGluon problem_type is using same names as amlb.data.DatasetType
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
 
+def run_autogluon_timeseries(dataset: Dataset, config: TaskConfig):
+    from frameworks.shared.caller import run_in_venv
+    dataset = deepcopy(dataset)
+    if not hasattr(dataset, 'timestamp_column'):
+        dataset.timestamp_column = None
+    if not hasattr(dataset, 'id_column'):
+        dataset.id_column = None
+    if not hasattr(dataset, 'forecast_range_in_steps'):
+        raise AttributeError("Unspecified `forecast_range_in_steps`.")
+
+    data = dict(
+        # train=dict(path=dataset.train.data_path('parquet')),
+        # test=dict(path=dataset.test.data_path('parquet')),
+        train=dict(path=dataset.train.path),
+        test=dict(path=dataset.test.path),
+        target=dict(
+            name=dataset.target.name,
+            classes=dataset.target.values
+        ),
+        problem_type=dataset.type.name,  # AutoGluon problem_type is using same names as amlb.data.DatasetType
+        timestamp_column=dataset.timestamp_column,
+        id_column=dataset.id_column,
+        forecast_range_in_steps=dataset.forecast_range_in_steps
+    )
+
+    return run_in_venv(__file__, "exec_ts.py",
+                       input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/AutoGluon/exec.py b/frameworks/AutoGluon/exec.py
index 7a7cfbb91..89c3372b4 100644
--- a/frameworks/AutoGluon/exec.py
+++ b/frameworks/AutoGluon/exec.py
@@ -4,6 +4,8 @@
 import warnings
 import sys
 import tempfile
+from typing import Union
+
 warnings.simplefilter("ignore")
 
 if sys.platform == 'darwin':
@@ -13,12 +15,13 @@
 import pandas as pd
 matplotlib.use('agg')  # no need for tk
 
-from autogluon.tabular import TabularPredictor
+from autogluon.tabular import TabularPredictor, TabularDataset
 from autogluon.core.utils.savers import save_pd, save_pkl
 import autogluon.core.metrics as metrics
 from autogluon.tabular.version import __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, zip_path
 
 log = logging.getLogger(__name__)
@@ -45,8 +48,22 @@ def run(dataset, config):
 
     is_classification = config.type == 'classification'
     training_params = {k: v for k, v in config.framework_params.items() if not k.startswith('_')}
+    presets = training_params.get("presets", [])
+    presets = presets if isinstance(presets, list) else [presets]
+    if preset_with_refit_full := (set(presets) & {"good_quality", "high_quality"}):
+        preserve = 0.9
+        preset = next(iter(preset_with_refit_full))
+        msg = (
+            f"Detected `{preset}` preset, reducing `max_runtime_seconds` "
+            f"from {config.max_runtime_seconds}s to "
+            f"{preserve * config.max_runtime_seconds}s to account for `refit_full` "
+            f"call after fit, which can take up to ~15% of total time. "
+            "See https://auto.gluon.ai/stable/api/autogluon.tabular.TabularPredictor.refit_full.html"
+        )
+        log.info(msg)
+        config.max_runtime_seconds = preserve * config.max_runtime_seconds
 
-    train, test = dataset.train.path, dataset.test.path
+    train_path, test_path = dataset.train.path, dataset.test.path
     label = dataset.target.name
     problem_type = dataset.problem_type
 
@@ -59,30 +76,49 @@ def run(dataset, config):
             path=models_dir,
             problem_type=problem_type,
         ).fit(
-            train_data=train,
+            train_data=train_path,
             time_limit=config.max_runtime_seconds,
             **training_params
         )
 
-    del train
+    log.info(f"Finished fit in {training.duration}s.")
+
+    # Persist model in memory that is going to be predicting to get correct inference latency
+    # max_memory=0.4 will be future default: https://github.com/autogluon/autogluon/pull/3338
+    predictor.persist_models('best', max_memory=0.4)
+
+    def inference_time_classification(data: Union[str, pd.DataFrame]):
+        return None, predictor.predict_proba(data, as_multiclass=True)
 
+    def inference_time_regression(data: Union[str, pd.DataFrame]):
+        return predictor.predict(data, as_pandas=False), None
+
+    infer = inference_time_classification if is_classification else inference_time_regression
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = pd.read_parquet(dataset.test.path)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+        log.info(f"Finished inference time measurements.")
+
+    test_data = TabularDataset(test_path)
+    with Timer() as predict:
+        predictions, probabilities = infer(test_data)
     if is_classification:
-        with Timer() as predict:
-            probabilities = predictor.predict_proba(test, as_multiclass=True)
         predictions = probabilities.idxmax(axis=1).to_numpy()
-    else:
-        with Timer() as predict:
-            predictions = predictor.predict(test, as_pandas=False)
-        probabilities = None
 
     prob_labels = probabilities.columns.values.astype(str).tolist() if probabilities is not None else None
+    log.info(f"Finished predict in {predict.duration}s.")
 
     _leaderboard_extra_info = config.framework_params.get('_leaderboard_extra_info', False)  # whether to get extra model info (very verbose)
     _leaderboard_test = config.framework_params.get('_leaderboard_test', False)  # whether to compute test scores in leaderboard (expensive)
     leaderboard_kwargs = dict(silent=True, extra_info=_leaderboard_extra_info)
     # Disabled leaderboard test data input by default to avoid long running computation, remove 7200s timeout limitation to re-enable
     if _leaderboard_test:
-        leaderboard_kwargs['data'] = test
+        leaderboard_kwargs['data'] = test_data
 
     leaderboard = predictor.leaderboard(**leaderboard_kwargs)
     with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', 1000):
@@ -105,7 +141,8 @@ def run(dataset, config):
                   models_count=num_models_trained,
                   models_ensemble_count=num_models_ensemble,
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,)
 
 
 def save_artifacts(predictor, leaderboard, config):
diff --git a/frameworks/AutoGluon/exec_ts.py b/frameworks/AutoGluon/exec_ts.py
new file mode 100644
index 000000000..ab7c4110f
--- /dev/null
+++ b/frameworks/AutoGluon/exec_ts.py
@@ -0,0 +1,172 @@
+import logging
+import os
+import shutil
+import warnings
+import sys
+import tempfile
+import numpy as np
+warnings.simplefilter("ignore")
+
+if sys.platform == 'darwin':
+    os.environ['OMP_NUM_THREADS'] = '1'
+
+import pandas as pd
+
+from autogluon.core.utils.savers import save_pd, save_pkl
+from autogluon.tabular import TabularDataset
+from autogluon.timeseries import TimeSeriesPredictor, TimeSeriesDataFrame
+from autogluon.timeseries.version import __version__
+
+from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.utils import Timer, zip_path
+
+log = logging.getLogger(__name__)
+
+
+def run(dataset, config):
+    log.info(f"\n**** AutoGluon TimeSeries [v{__version__}] ****\n")
+
+    timestamp_column = dataset.timestamp_column
+    id_column = dataset.id_column
+    prediction_length = dataset.forecast_range_in_steps
+
+    eval_metric = get_eval_metric(config)
+    label = dataset.target.name
+    time_limit = config.max_runtime_seconds
+
+    training_params = {k: v for k, v in config.framework_params.items() if not k.startswith('_')}
+
+    train_data, test_data = load_data(train_path=dataset.train.path,
+                                      test_path=dataset.test.path,
+                                      timestamp_column=timestamp_column,
+                                      id_column=id_column)
+    test_data_past = test_data.copy().slice_by_timestep(slice(None, -prediction_length))
+
+    predictor_path = tempfile.mkdtemp() + os.sep
+    with Timer() as training:
+        predictor = TimeSeriesPredictor(
+            target=label,
+            path=predictor_path,
+            prediction_length=prediction_length,
+            eval_metric=eval_metric,
+        )
+        predictor.fit(
+            train_data=train_data,
+            time_limit=time_limit,
+            **training_params,
+        )
+
+    with Timer() as predict:
+        predictions = predictor.predict(test_data_past)
+    log.info(predictions)
+
+    predictions_only = predictions['mean'].values
+    test_data_future = test_data.copy().slice_by_timestep(slice(-prediction_length, None))
+    truth_only = test_data_future[label].values
+
+    log.info(predictions_only)
+    log.info(truth_only)
+
+    leaderboard = predictor.leaderboard(test_data, silent=True)
+
+    with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', 1000):
+        log.info(leaderboard)
+
+    num_models_trained = len(leaderboard)
+
+    save_artifacts(predictor=predictor, leaderboard=leaderboard, config=config)
+    shutil.rmtree(predictor.path, ignore_errors=True)
+
+    quantiles = predictions.drop(columns=['mean']).reset_index(drop=True)
+    period_length = 1 # TODO: This period length could be adapted to the Dataset, but then we need to pass this information as well. As of now this works.
+
+    # we aim to calculate the mean period error from the past for each sequence: 1/N sum_{i=1}^N |x(t_i) - x(t_i - T)|
+    # 1. retrieve item_ids for each sequence/item
+    #dataset..X /. y
+    item_ids, inverse_item_ids = np.unique(test_data.reset_index()["item_id"].squeeze().to_numpy(), return_index=False, return_inverse=True)
+    # 2. capture sequences in a list
+    y_past = [test_data[label].squeeze().to_numpy()[inverse_item_ids == i][:-prediction_length] for i in range(len(item_ids))]
+    # 3. calculate period error per sequence
+    y_past_period_error = [np.abs(y_past_item[period_length:] - y_past_item[:-period_length]).mean() for y_past_item in y_past]
+    # 4. repeat period error for each sequence, to save one for each element
+    y_past_period_error_rep = np.repeat(y_past_period_error, prediction_length)
+
+    optional_columns = quantiles
+    optional_columns = optional_columns.assign(y_past_period_error=y_past_period_error_rep)
+
+    return result(output_file=config.output_predictions_file,
+                  predictions=predictions_only,
+                  truth=truth_only,
+                  probabilities=None,
+                  probabilities_labels=None,
+                  target_is_encoded=False,
+                  models_count=num_models_trained,
+                  training_duration=training.duration,
+                  predict_duration=predict.duration,
+                  optional_columns=optional_columns)
+
+def load_data(train_path, test_path, timestamp_column, id_column):
+
+    train_df = pd.read_csv(
+        train_path,
+        parse_dates=[timestamp_column],
+    )
+
+    train_data = TimeSeriesDataFrame.from_data_frame(
+        train_df,
+        id_column=id_column,
+        timestamp_column=timestamp_column,
+    )
+
+    test_df = pd.read_csv(
+        test_path,
+        parse_dates=[timestamp_column],
+    )
+
+    test_data = TimeSeriesDataFrame.from_data_frame(
+        test_df,
+        id_column=id_column,
+        timestamp_column=timestamp_column,
+    )
+
+    return train_data, test_data
+
+
+def get_eval_metric(config):
+    # TODO: Support more metrics
+    metrics_mapping = dict(
+        mape="MAPE",
+        smape="sMAPE",
+        mase="MASE",
+        mse="MSE",
+        rmse="RMSE",
+    )
+
+    eval_metric = metrics_mapping[config.metric] if config.metric in metrics_mapping else None
+    if eval_metric is None:
+        log.warning("Performance metric %s not supported.", config.metric)
+    return eval_metric
+
+
+def save_artifacts(predictor, leaderboard, config):
+    artifacts = config.framework_params.get('_save_artifacts', ['leaderboard'])
+    try:
+        if 'leaderboard' in artifacts:
+            leaderboard_dir = output_subdir("leaderboard", config)
+            save_pd.save(path=os.path.join(leaderboard_dir, "leaderboard.csv"), df=leaderboard)
+
+        if 'info' in artifacts:
+            ag_info = predictor.info()
+            info_dir = output_subdir("info", config)
+            save_pkl.save(path=os.path.join(info_dir, "info.pkl"), object=ag_info)
+
+        if 'models' in artifacts:
+            shutil.rmtree(os.path.join(predictor.path, "utils"), ignore_errors=True)
+            models_dir = output_subdir("models", config)
+            zip_path(predictor.path, os.path.join(models_dir, "models.zip"))
+    except Exception:
+        log.warning("Error when saving artifacts.", exc_info=True)
+
+
+if __name__ == '__main__':
+    call_run(run)
diff --git a/frameworks/AutoGluon/setup.sh b/frameworks/AutoGluon/setup.sh
index 6e9a2a76f..967c9d40c 100755
--- a/frameworks/AutoGluon/setup.sh
+++ b/frameworks/AutoGluon/setup.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+
 HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/awslabs/autogluon.git"}
@@ -9,34 +10,37 @@ fi
 
 # creating local venv
 . ${HERE}/../shared/setup.sh ${HERE} true
-#if [[ -x "$(command -v apt-get)" ]]; then
-#    SUDO apt-get install -y libomp-dev
+
+# Below fixes seg fault on MacOS due to bug in libomp: https://github.com/awslabs/autogluon/issues/1442
 if [[ -x "$(command -v brew)" ]]; then
-    brew install libomp
+    wget https://raw.githubusercontent.com/Homebrew/homebrew-core/fb8323f2b170bd4ae97e1bac9bf3e2983af3fdb0/Formula/libomp.rb -P "${HERE}/lib"
+    brew install "${HERE}/lib/libomp.rb"
 fi
 
 PIP install --upgrade pip
 PIP install --upgrade setuptools wheel
-PIP install "mxnet<2.0.0"
-PIP install "scikit-learn-intelex<2021.3"
 
 if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
+    PIP install --no-cache-dir -U "${PKG}"
+    PIP install --no-cache-dir -U "${PKG}.tabular[skex]"
 elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+    PIP install --no-cache-dir -U "${PKG}==${VERSION}"
+    PIP install --no-cache-dir -U "${PKG}.tabular[skex]==${VERSION}"
 else
     TARGET_DIR="${HERE}/lib/${PKG}"
     rm -Rf ${TARGET_DIR}
     git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
     cd ${TARGET_DIR}
-    PIP install -e core/
-    PIP install -e features/
-    PIP install -e tabular/[all]
-    PIP install -e mxnet/
-    PIP install -e extra/
-    PIP install -e text/
-    PIP install -e vision/
-    PIP install -e autogluon/
+    PY_EXEC_NO_ARGS="$(cut -d' ' -f1 <<<"$py_exec")"
+    PY_EXEC_DIR=$(dirname "$PY_EXEC_NO_ARGS")
+    env PATH="$PY_EXEC_DIR:$PATH" bash -c ./full_install.sh
+    PIP install -e tabular/[skex]
 fi
 
-PY -c "from autogluon.tabular.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+if [[ ${MODULE} == "timeseries" ]]; then
+    PY -c "from autogluon.timeseries.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+    # TODO: GPU version install
+    PIP install "mxnet<2.0"
+else
+    PY -c "from autogluon.tabular.version import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+fi
diff --git a/frameworks/DecisionTree/exec.py b/frameworks/DecisionTree/exec.py
index 0ce133c20..f46eb01c4 100644
--- a/frameworks/DecisionTree/exec.py
+++ b/frameworks/DecisionTree/exec.py
@@ -25,9 +25,12 @@ def run(dataset: Dataset, config: TaskConfig):
 
     with Timer() as training:
         predictor.fit(X_train, y_train)
+    log.info(f"Finished fit in {training.duration}s.")
+
     with Timer() as predict:
         predictions = predictor.predict(X_test)
     probabilities = predictor.predict_proba(X_test) if is_classification else None
+    log.info(f"Finished predict in {predict.duration}s.")
 
     save_predictions(dataset=dataset,
                      output_file=config.output_predictions_file,
diff --git a/frameworks/GAMA/__init__.py b/frameworks/GAMA/__init__.py
index f660e2f8f..750f5e74e 100644
--- a/frameworks/GAMA/__init__.py
+++ b/frameworks/GAMA/__init__.py
@@ -23,6 +23,8 @@ def run(dataset: Dataset, config: TaskConfig):
             y=dataset.test.y
         ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/GAMA/exec.py b/frameworks/GAMA/exec.py
index e0880bf34..9d17061db 100644
--- a/frameworks/GAMA/exec.py
+++ b/frameworks/GAMA/exec.py
@@ -2,6 +2,9 @@
 import os
 import sys
 import tempfile as tmp
+from typing import Union
+
+import pandas as pd
 
 if sys.platform == 'darwin':
     os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'
@@ -18,7 +21,8 @@
 from gama.data_loading import file_to_pandas
 from gama import GamaClassifier, GamaRegressor, __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, touch
 
 
@@ -72,23 +76,34 @@ def run(dataset, config):
     gama_automl = estimator(**kwargs)
 
     X_train, y_train = dataset.train.X, dataset.train.y
-    # data = file_to_pandas(dataset.train.path, encoding='utf-8')
-    # X_train, y_train = data.loc[:, data.columns != dataset.target], data.loc[:, dataset.target]
-
-    with Timer() as training_timer:
+    with Timer() as training:
         gama_automl.fit(X_train, y_train)
+    log.info(f"Finished fit in {training.duration}s.")
 
-    log.info('Predicting on the test set.')
-    X_test, y_test = dataset.test.X, dataset.test.y
-    # data = file_to_pandas(dataset.test.path, encoding='utf-8')
-    # X_test, y_test = data.loc[:, data.columns != dataset.target], data.loc[:, dataset.target]
 
-    with Timer() as predict_timer:
+    log.info('Predicting on the test set.')
+    def infer(data: Union[str, pd.DataFrame]):
+        test_data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = gama_automl.predict_proba if is_classification else gama_automl.predict
+        return predict_fn(test_data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+        log.info(f"Finished inference time measurements.")
+
+    with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y
         predictions = gama_automl.predict(X_test)
+    log.info(f"Finished predict in {predict.duration}s.")
+
+    probabilities = None
     if is_classification:
         probabilities = gama_automl.predict_proba(X_test)
-    else:
-        probabilities = None
 
     return result(
         output_file=config.output_predictions_file,
@@ -97,8 +112,9 @@ def run(dataset, config):
         truth=y_test,
         target_is_encoded=False,
         models_count=len(gama_automl._final_pop),
-        training_duration=training_timer.duration,
-        predict_duration=predict_timer.duration
+        training_duration=training.duration,
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/H2OAutoML/__init__.py b/frameworks/H2OAutoML/__init__.py
index f8faf87a5..596513181 100644
--- a/frameworks/H2OAutoML/__init__.py
+++ b/frameworks/H2OAutoML/__init__.py
@@ -8,24 +8,17 @@ def setup(*args, **kwargs):
     call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
 
 
-# def version():
-#     from frameworks.shared.caller import run_cmd_in_venv
-#     out, err = run_cmd_in_venv(__file__, """{py} -c "from h2o import __version__; print(__version__)" | grep "^\d\." """)
-#     if err:
-#         raise ValueError(err)
-#     return out
-
-
 def run(dataset: Dataset, config: TaskConfig):
     from frameworks.shared.caller import run_in_venv
-
     data = dict(
         train=dict(path=dataset.train.path),
         test=dict(path=dataset.test.path),
         target=dict(index=dataset.target.index),
-        domains=dict(cardinalities=[0 if f.values is None else len(f.values) for f in dataset.features])
+        domains=dict(cardinalities=[0 if f.values is None else len(f.values) for f in dataset.features]),
+        format=dataset.train.format,
     )
-
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt=dataset.train.format, with_labels=True)
     config.ext.monitoring = rconfig().monitoring
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/H2OAutoML/exec.py b/frameworks/H2OAutoML/exec.py
index de215b172..698ad83b6 100644
--- a/frameworks/H2OAutoML/exec.py
+++ b/frameworks/H2OAutoML/exec.py
@@ -1,6 +1,8 @@
 import contextlib
 import logging
 import os
+import pathlib
+
 import psutil
 import re
 
@@ -10,7 +12,8 @@
 import h2o
 from h2o.automl import H2OAutoML
 
-from frameworks.shared.callee import FrameworkError, call_run, output_subdir, result
+from frameworks.shared.callee import FrameworkError, call_run, output_subdir, result, \
+    measure_inference_times
 from frameworks.shared.utils import Monitoring, Namespace as ns, Timer, clean_dir, touch, zip_path
 
 log = logging.getLogger(__name__)
@@ -88,6 +91,10 @@ def run(dataset, config):
         test = h2o.import_file(dataset.test.path, destination_frame=frame_name('test', config), **import_kwargs)
         # test.impute(method='mean')
 
+        if config.type == 'classification' and dataset.format == 'csv':
+            train[dataset.target.index] = train[dataset.target.index].asfactor()
+            test[dataset.target.index] = test[dataset.target.index].asfactor()
+
         log.info("Running model on task %s, fold %s.", config.name, config.fold)
         log.debug("Running H2O AutoML with a maximum time of %ss on %s core(s), optimizing %s.",
                   config.max_runtime_seconds, config.cores, sort_metric)
@@ -101,18 +108,33 @@ def run(dataset, config):
                                            check_on_exit=True,
                                            verbosity=config.ext.monitoring.verbosity)
                    if config.framework_params.get('_monitor_backend', False)
-                   # else contextlib.nullcontext  # Py 3.7+ only
-                   else contextlib.contextmanager(lambda: (_ for _ in (0,)))()
+                   else contextlib.nullcontext()  # Py 3.7+ only
+                   # else contextlib.contextmanager(lambda: (_ for _ in (0,)))()
                    )
         with Timer() as training:
             with monitor:
                 aml.train(y=dataset.target.index, training_frame=train)
+        log.info(f"Finished fit in {training.duration}s.")
+
 
         if not aml.leader:
             raise FrameworkError("H2O could not produce any model in the requested time.")
 
+        def infer(path: str):
+            filename = pathlib.Path(path).name
+            # H2O can't do inference on single row arff, it needs columns explicitly:
+            # https://github.com/h2oai/h2o-3/issues/15572
+            batch = h2o.import_file(path, col_names=train.names, destination_frame=frame_name(filename, config), **import_kwargs)
+            return aml.predict(batch)
+
+        inference_times = {}
+        if config.measure_inference_time:
+            inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+            log.info(f"Finished inference time measurements.")
+
         with Timer() as predict:
             preds = aml.predict(test)
+        log.info(f"Finished predict in {predict.duration}s.")
 
         preds = extract_preds(preds, test, dataset=dataset)
         save_artifacts(aml, dataset=dataset, config=config)
@@ -125,7 +147,8 @@ def run(dataset, config):
             probabilities_labels=preds.probabilities_labels,
             models_count=len(aml.leaderboard),
             training_duration=training.duration,
-            predict_duration=predict.duration
+            predict_duration=predict.duration,
+            inference_times=inference_times,
         )
 
     finally:
diff --git a/frameworks/MLNet/exec.py b/frameworks/MLNet/exec.py
index 0dbdb86a5..c2cbb6074 100644
--- a/frameworks/MLNet/exec.py
+++ b/frameworks/MLNet/exec.py
@@ -61,6 +61,7 @@ def run(dataset: Dataset, config: TaskConfig):
 
         with Timer() as training:
             run_cmd(cmd)
+        log.info(f"Finished fit in {training.duration}s.")
 
         train_result_json = os.path.join(output_dir, '{}.mbconfig'.format(config.fold))
         if not os.path.exists(train_result_json):
@@ -75,8 +76,9 @@ def run(dataset: Dataset, config: TaskConfig):
             # predict
             predict_cmd = (f"{mlnet} predict --task-type {config.type}"
                            f" --model {model_path} --dataset {test_dataset_path} --label-col {dataset.target.name} > {output_prediction_path}")
-            with Timer() as prediction:
+            with Timer() as predict:
                 run_cmd(predict_cmd)
+            log.info(f"Finished predict in {predict.duration}s.")
             if config.type == 'classification':
                 prediction_df = pd.read_csv(output_prediction_path, dtype={'PredictedLabel': 'object'})
 
@@ -101,7 +103,7 @@ def run(dataset: Dataset, config: TaskConfig):
             return dict(
                     models_count=models_count,
                     training_duration=training.duration,
-                    predict_duration=prediction.duration,
+                    predict_duration=predict.duration,
                 )
     finally:
         if 'logs' in artifacts:
diff --git a/frameworks/MLPlan/exec.py b/frameworks/MLPlan/exec.py
index 839d94d23..3cffb3eec 100644
--- a/frameworks/MLPlan/exec.py
+++ b/frameworks/MLPlan/exec.py
@@ -79,6 +79,7 @@ def run(dataset, config):
 
         with Timer() as training:
             run_cmd(cmd, _live_output_=True)
+        log.info(f"Finished fit in {training.duration}s.")
 
     with open(statistics_file, 'r') as f:
         stats = json.load(f)
diff --git a/frameworks/MLPlan/setup.sh b/frameworks/MLPlan/setup.sh
index f80e0d53c..5359f4cb2 100755
--- a/frameworks/MLPlan/setup.sh
+++ b/frameworks/MLPlan/setup.sh
@@ -26,7 +26,7 @@ rm -Rf ${TARGET_DIR}
 
 mkdir -p $DOWNLOAD_DIR
 echo "Download ML-Plan from extern"
-wget -q https://download.mlplan.org/version/$VERSION -O $DOWNLOAD_DIR/$MLPLAN_ARC
+wget -q --no-check-certificate https://download.mlplan.org/version/$VERSION -O $DOWNLOAD_DIR/$MLPLAN_ARC
 echo "Download finished. Now unzip the downloaded file."
 unzip $DOWNLOAD_DIR/$MLPLAN_ARC -d $TARGET_DIR
 
diff --git a/frameworks/RandomForest/__init__.py b/frameworks/RandomForest/__init__.py
index a25cbeee7..5d1f0aa49 100644
--- a/frameworks/RandomForest/__init__.py
+++ b/frameworks/RandomForest/__init__.py
@@ -23,8 +23,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/RandomForest/exec.py b/frameworks/RandomForest/exec.py
index 77bdc99ef..4d7eaa0f7 100644
--- a/frameworks/RandomForest/exec.py
+++ b/frameworks/RandomForest/exec.py
@@ -3,6 +3,8 @@
 import tempfile as tmp
 from typing import List
 
+import pandas as pd
+
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
 os.environ['OMP_NUM_THREADS'] = '1'
 os.environ['OPENBLAS_NUM_THREADS'] = '1'
@@ -12,7 +14,7 @@
 import sklearn
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
 
-from frameworks.shared.callee import call_run, result
+from frameworks.shared.callee import call_run, result, measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(os.path.basename(__file__))
@@ -81,10 +83,28 @@ def run(dataset, config):
             else:
                 # https://stackoverflow.com/questions/42757892/how-to-use-warm-start/42763502
                 rf.n_estimators += step_size
+    log.info(f"Finished fit in {training.duration}s.")
+
 
     with Timer() as predict:
         predictions = rf.predict(X_test)
     probabilities = rf.predict_proba(X_test) if is_classification else None
+    log.info(f"Finished predict in {predict.duration}s.")
+
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return rf.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+    log.info(f"Finished inference time measurements.")
+
 
     return result(output_file=config.output_predictions_file,
                   predictions=predictions,
@@ -93,7 +113,8 @@ def run(dataset, config):
                   target_is_encoded=encode,
                   models_count=len(rf),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,)
 
 
 if __name__ == '__main__':
diff --git a/frameworks/TPOT/__init__.py b/frameworks/TPOT/__init__.py
index 9828c6473..10b5ddb5f 100644
--- a/frameworks/TPOT/__init__.py
+++ b/frameworks/TPOT/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     def process_results(results):
         if isinstance(results.probabilities, str) and results.probabilities == "predictions":
diff --git a/frameworks/TPOT/exec.py b/frameworks/TPOT/exec.py
index ce70cb7f9..4ec2d45e0 100644
--- a/frameworks/TPOT/exec.py
+++ b/frameworks/TPOT/exec.py
@@ -4,6 +4,9 @@
 import sys
 import tempfile as tmp
 
+import pandas as pd
+from numpy.random import default_rng
+
 if sys.platform == 'darwin':
     os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
@@ -13,7 +16,8 @@
 
 from tpot import TPOTClassifier, TPOTRegressor, __version__
 
-from frameworks.shared.callee import call_run, output_subdir, result
+from frameworks.shared.callee import call_run, output_subdir, result, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, is_sparse
 
 
@@ -61,18 +65,44 @@ def run(dataset, config):
 
     with Timer() as training:
         tpot.fit(X_train, y_train)
+    log.info(f"Finished fit in {training.duration}s.")
+
+
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        if is_classification:
+            try:
+                return tpot.predict_proba(data)
+            except (RuntimeError, AttributeError):
+                return tpot.predict(data)
+        return tpot.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        log.info("TPOT inference time measurements exclude preprocessing time of AMLB.")
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer, [
+                (1, dataset.test.X[default_rng(seed=i).integers(len(dataset.test.X)), :].reshape(1, -1))
+                for i in range(100)
+            ],
+        )
+        log.info(f"Finished inference time measurements.")
 
     log.info('Predicting on the test set.')
-    X_test = dataset.test.X
     y_test = dataset.test.y
     with Timer() as predict:
+        X_test = dataset.test.X
         predictions = tpot.predict(X_test)
+
     try:
         probabilities = tpot.predict_proba(X_test) if is_classification else None
-    except RuntimeError:
-        # TPOT throws a RuntimeError if the optimized pipeline does not support `predict_proba`.
+    except (RuntimeError, AttributeError):
+        # TPOT throws a RuntimeError or AttributeError if the optimized pipeline
+        # does not support `predict_proba` (which one depends on the version).
         probabilities = "predictions"  # encoding is handled by caller in `__init__.py`
 
+    log.info(f"Finished predict in {predict.duration}s.")
     save_artifacts(tpot, config)
 
     return result(output_file=config.output_predictions_file,
@@ -82,7 +112,9 @@ def run(dataset, config):
                   target_is_encoded=is_classification,
                   models_count=len(tpot.evaluated_individuals_),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,
+                  )
 
 
 def save_artifacts(estimator, config):
diff --git a/frameworks/TunedRandomForest/__init__.py b/frameworks/TunedRandomForest/__init__.py
index 561678497..b97439508 100644
--- a/frameworks/TunedRandomForest/__init__.py
+++ b/frameworks/TunedRandomForest/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         test=dict(
             X=X_test,
             y=y_test
-        )
+        ),
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet", scikit_safe=True)
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/TunedRandomForest/exec.py b/frameworks/TunedRandomForest/exec.py
index 7c7a7dc15..c724487d0 100644
--- a/frameworks/TunedRandomForest/exec.py
+++ b/frameworks/TunedRandomForest/exec.py
@@ -21,7 +21,7 @@
 import sklearn
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
 
-from frameworks.shared.callee import call_run, result
+from frameworks.shared.callee import call_run, result, measure_inference_times
 from frameworks.shared.utils import Timer
 from custom_validate import cross_validate
 
@@ -211,6 +211,19 @@ def run(dataset, config):
         predictions = rf.predict(X_test)
     probabilities = rf.predict_proba(X_test) if is_classification else None
 
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return rf.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+
     return result(
         output_file=config.output_predictions_file,
         predictions=predictions,
@@ -219,7 +232,8 @@ def run(dataset, config):
         target_is_encoded=is_classification,
         models_count=len(rf),
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/autosklearn/__init__.py b/frameworks/autosklearn/__init__.py
index 3e31b6d64..637a81491 100644
--- a/frameworks/autosklearn/__init__.py
+++ b/frameworks/autosklearn/__init__.py
@@ -10,19 +10,23 @@ def setup(*args, **kwargs):
 def run(dataset: Dataset, config: TaskConfig):
     from frameworks.shared.caller import run_in_venv
 
-    X_train, X_test = dataset.train.X_enc, dataset.test.X_enc
-    y_train, y_test = unsparsify(dataset.train.y_enc, dataset.test.y_enc)
     data = dict(
         train=dict(
-            X=X_train,
-            y=y_train
+            X=dataset.train.X,
+            y=dataset.train.y,
+            X_enc=dataset.train.X_enc,
+            y_enc=unsparsify(dataset.train.y_enc),
         ),
         test=dict(
-            X=X_test,
-            y=y_test
+            X=dataset.test.X,
+            y=dataset.test.y,
+            X_enc=dataset.test.X_enc,
+            y_enc=unsparsify(dataset.test.y_enc),
         ),
-        predictors_type=['Numerical' if p.is_numerical() else 'Categorical' for p in dataset.predictors]
+        predictors_type=['Numerical' if p.is_numerical() else 'Categorical' for p in dataset.predictors],
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
 
     return run_in_venv(__file__, "exec.py",
                        input_data=data, dataset=dataset, config=config)
diff --git a/frameworks/autosklearn/exec.py b/frameworks/autosklearn/exec.py
index 11690e09c..0ebbe1edf 100644
--- a/frameworks/autosklearn/exec.py
+++ b/frameworks/autosklearn/exec.py
@@ -4,6 +4,10 @@
 import shutil
 import tempfile as tmp
 import warnings
+from typing import Union
+
+import pandas as pd
+from numpy.random import default_rng
 
 os.environ['JOBLIB_TEMP_FOLDER'] = tmp.gettempdir()
 os.environ['OMP_NUM_THREADS'] = '1'
@@ -15,7 +19,8 @@
 import autosklearn.metrics as metrics
 from packaging import version
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer, system_memory_mb, walk_apply, zip_path
 
 log = logging.getLogger(__name__)
@@ -62,8 +67,9 @@ def run(dataset, config):
     )
     log.info("Environment: %s", os.environ)
 
-    X_train = dataset.train.X
-    y_train = dataset.train.y
+    use_pandas = askl_version >= version.parse("0.15")
+    X_train = dataset.train.X if use_pandas else dataset.train.X_enc
+    y_train = dataset.train.y if use_pandas else dataset.train.y_enc
     predictors_type = dataset.predictors_type
     log.debug("predictors_type=%s", predictors_type)
 
@@ -118,6 +124,10 @@ def run(dataset, config):
     else:
         fit_extra_params['metric'] = perf_metric
 
+    if not use_pandas:
+        fit_extra_params["feat_type"] = predictors_type
+
+
     constr_params["time_left_for_this_task"] = config.max_runtime_seconds
     constr_params["n_jobs"] = n_jobs
     constr_params["seed"] = config.seed
@@ -128,26 +138,48 @@ def run(dataset, config):
 
     auto_sklearn = estimator(**constr_params, **training_params)
     with Timer() as training:
-        auto_sklearn.fit(X_train, y_train, feat_type=predictors_type, **fit_extra_params)
+        auto_sklearn.fit(X_train, y_train, **fit_extra_params)
+    log.info(f"Finished fit in {training.duration}s.")
+
+    def infer(data: Union[str, pd.DataFrame]):
+        test_data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = auto_sklearn.predict_proba if is_classification else auto_sklearn.predict
+        return predict_fn(test_data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        test_data = dataset.test.X if use_pandas else dataset.test.X_enc
+        def sample_one_test_row(seed: int):
+            if use_pandas:
+                return test_data.sample(1, random_state=seed)
+            return test_data[default_rng(seed=seed).integers(len(test_data)), :]
+
+        inference_times["df"] = measure_inference_times(
+            infer, [(1, sample_one_test_row(seed=i)) for i in range(100)],
+        )
+        log.info(f"Finished inference time measurements.")
 
     # Convert output to strings for classification
     log.info("Predicting on the test set.")
-    X_test = dataset.test.X
-    y_test = dataset.test.y
     with Timer() as predict:
+        X_test = dataset.test.X if use_pandas else dataset.test.X_enc
         predictions = auto_sklearn.predict(X_test)
     probabilities = auto_sklearn.predict_proba(X_test) if is_classification else None
+    log.info(f"Finished predict in {predict.duration}s.")
 
     save_artifacts(auto_sklearn, config)
 
     return result(output_file=config.output_predictions_file,
                   predictions=predictions,
-                  truth=y_test,
+                  truth=dataset.test.y if use_pandas else dataset.test.y_enc,
                   probabilities=probabilities,
-                  target_is_encoded=is_classification,
+                  target_is_encoded=is_classification and not use_pandas,
                   models_count=len(auto_sklearn.get_models_with_weights()),
                   training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  inference_times=inference_times,
+                  )
 
 
 def save_artifacts(estimator, config):
diff --git a/frameworks/autosklearn/requirements.txt b/frameworks/autosklearn/requirements.txt
index 6d4ac2d7d..9e7bca5c3 100644
--- a/frameworks/autosklearn/requirements.txt
+++ b/frameworks/autosklearn/requirements.txt
@@ -1,3 +1,3 @@
-packaging
+packaging<22.0
 openml
-scipy>=0.14.1,<1.7.0
+pandas<2.0  # https://github.com/automl/auto-sklearn/issues/1672
diff --git a/frameworks/autosklearn/setup.sh b/frameworks/autosklearn/setup.sh
index f0099c3e3..63edcf04f 100755
--- a/frameworks/autosklearn/setup.sh
+++ b/frameworks/autosklearn/setup.sh
@@ -36,6 +36,7 @@ else
     git clone  --recurse-submodules --shallow-submodules ${DEPTH} ${REPO} ${TARGET_DIR}
     cd ${TARGET_DIR}
     git checkout "${COMMIT}"
+    git submodule update --init --recursive
     cd ${HERE}
     PIP install -U -e ${TARGET_DIR}
 fi
diff --git a/frameworks/autoxgboost/__init__.py b/frameworks/autoxgboost/__init__.py
deleted file mode 100644
index e77eab6f7..000000000
--- a/frameworks/autoxgboost/__init__.py
+++ /dev/null
@@ -1,10 +0,0 @@
-from amlb.utils import call_script_in_same_dir
-
-
-def setup(*args, **kwargs):
-    call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
-
-
-def run(*args, **kwargs):
-    from .exec import run
-    return run(*args, **kwargs)
diff --git a/frameworks/autoxgboost/exec.R b/frameworks/autoxgboost/exec.R
deleted file mode 100644
index 08c54f90f..000000000
--- a/frameworks/autoxgboost/exec.R
+++ /dev/null
@@ -1,47 +0,0 @@
-library(mlr)
-library(autoxgboost)
-library(farff)
-
-run <- function(train_file, test_file, target.index, type, output_predictions_file, cores, time.budget, meta_results_file) {
-  train <- farff::readARFF(train_file)
-  colnames(train) <- make.names(colnames(train))
-  target <- colnames(train)[target.index]
-
-  if (type == "classification") {
-    train <- makeClassifTask(data = train, target = target)
-    lrn <- makeLearner("classif.autoxgboost", time.budget = time.budget, nthread = cores, predict.type = "prob")
-  } else if (type == "regression") {
-    train <- makeRegrTask(data = train, target = target)
-    lrn <- makeLearner("regr.autoxgboost", time.budget = time.budget, nthread = cores)
-  } else {
-    stop("Task type not supported!")
-  }
-
-  mod <- NULL
-  preds <- NULL
-  training <- function() mod <<- train(lrn, train)
-  prediction <- function() preds <<- predict(mod, newdata = test)$data
-
-  train_duration <- system.time(training())[['elapsed']]
-
-  test <- farff::readARFF(test_file)
-  colnames(test) <- make.names(colnames(test))
-  predict_duration <- system.time(prediction())[['elapsed']]
-
-  preds <- preds[c(2:ncol(preds), 1)]
-  names(preds)[names(preds) == "response"] <- "predictions"
-
-  if (type == "classification") {
-    names(preds) <- sub("^prob.", "", names(preds))
-  }
-  # FIXME: label encoding for predictions and truth?
-
-  write.csv(preds, file = output_predictions_file, row.names = FALSE)
-
-  meta_results <- data.frame(key=c("training_duration", "predict_duration"),
-                             value=c(train_duration, predict_duration))
-  write.csv(meta_results, file = meta_results_file, row.names = FALSE)
-}
-
-# args = commandArgs(trailingOnly=TRUE)
-# run(args[1], args[2], args[3], as.integer(args[4]))
diff --git a/frameworks/autoxgboost/exec.py b/frameworks/autoxgboost/exec.py
deleted file mode 100644
index a4ac54799..000000000
--- a/frameworks/autoxgboost/exec.py
+++ /dev/null
@@ -1,49 +0,0 @@
-import logging
-import os
-
-from amlb.benchmark import TaskConfig
-from amlb.data import Dataset
-from amlb.datautils import read_csv
-from amlb.utils import dir_of, run_cmd
-
-log = logging.getLogger(__name__)
-
-
-def run(dataset: Dataset, config: TaskConfig):
-    #TODO: use rpy2 instead? not necessary here though as the call is very simple
-    log.info(f"\n**** Autoxgboost (R) [{config.framework_version}] ****\n")
-
-    is_classification = config.type == 'classification'
-
-    here = dir_of(__file__)
-
-    meta_results_file = os.path.join(config.output_dir, "meta_results.csv")
-    run_cmd(("Rscript --vanilla -e \""
-             ".libPaths('{package_directory}'); source('{script}'); "
-             "run('{train}', '{test}', target.index = {target_index}, '{type}', '{output}', {cores},"
-             " time.budget = {time_budget}, meta_results_file='{meta_results}')"
-             "\"").format(
-        package_directory=os.path.join(here, "lib"),
-        script=os.path.join(here, 'exec.R'),
-        train=dataset.train.path,
-        test=dataset.test.path,
-        target_index=dataset.target.index+1,
-        type=config.type,
-        output=config.output_predictions_file,
-        cores=config.cores,
-        time_budget=config.max_runtime_seconds,
-        meta_results=meta_results_file
-    ), _live_output_=True)
-
-    log.info("Predictions saved to %s", config.output_predictions_file)
-
-    meta_results = read_csv(meta_results_file)
-    return dict(
-        training_duration=meta_result(meta_results, 'training_duration'),
-        predict_duration=meta_result(meta_results, 'predict_duration')
-    )
-
-
-def meta_result(df, key):
-    return df.loc[df['key'] == key, 'value'].squeeze()
-
diff --git a/frameworks/autoxgboost/requirements.txt b/frameworks/autoxgboost/requirements.txt
deleted file mode 100644
index 9ac4fbb1b..000000000
--- a/frameworks/autoxgboost/requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-rpy2==2.3.0
\ No newline at end of file
diff --git a/frameworks/autoxgboost/setup.sh b/frameworks/autoxgboost/setup.sh
deleted file mode 100755
index f73244502..000000000
--- a/frameworks/autoxgboost/setup.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/usr/bin/env bash
-HERE=$(dirname "$0")
-VERSION=${1:-"stable"}
-REPO=${2:-"ja-thomas/autoxgboost"}
-
-# Version can be specified as 'stable', 'latest', a branch or a commit hash (indicated by starting with '#')
-if [[ "$VERSION" == "latest" || "$VERSION" == "stable" ]]; then
-  VERSION="master"
-fi
-
-if [[ "$VERSION" =~ ^# ]]; then
-  VERSION="${VERSION:1}"
-else
-  # if VERSION is not a hash, it should be a branch (or a format which is not (officially) supported)
-  COMMIT=$(git ls-remote "https://github.com/${REPO}" | grep "refs/heads/${VERSION}" | cut -f 1)
-  if [[ -z $COMMIT ]]; then
-    echo "Could not resolve version ${VERSION}. It is not a branch on https://github.com/${REPO}."
-    echo "Continuing setup, install_github will try to resolve 'ref=${VERSION}'."
-  else
-    VERSION=$COMMIT
-  fi
-fi
-
-. ${HERE}/../shared/setup.sh "${HERE}"
-if [[ -x "$(command -v apt-get)" ]]; then
-  SUDO apt-get update
-  SUDO apt-get install -y software-properties-common dirmngr
-  SUDO apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
-  SUDO add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
-  SUDO apt-get update
-  SUDO apt-get install -y r-base r-base-dev
-  SUDO apt-get install -y libgdal-dev libproj-dev
-  SUDO apt-get install -y libssl-dev libcurl4-openssl-dev
-  SUDO apt-get install -y libcairo2-dev libudunits2-dev
-fi
-
-# We install the packages to a subdirectory in the framework folder, similar to venvs for Python, because:
-# - we want to be able to install different package versions for different frameworks in one local installation
-# - the default package directory is not always writeable (e.g. on Github CI)
-LIB="${HERE}/lib/"
-mkdir ${LIB}
-
-Rscript -e 'options(install.packages.check.source="no"); install.packages(c("remotes", "mlr", "mlrMBO", "mlrCPO", "farff", "GenSA", "rgenoud", "xgboost"), repos="https://cloud.r-project.org/", lib="'"${LIB}"'")'
-Rscript -e '.libPaths("'"${LIB}"'"); remotes::install_github("'"${REPO}"'", ref="'"${VERSION}"'", lib="'"${LIB}/"'")'
-
-OFFICIAL_VERSION=$(Rscript -e '.libPaths("'"${LIB}"'"); packageVersion("autoxgboost")' | awk '{print $2}' | sed "s/[‘’]//g")
-echo "${OFFICIAL_VERSION}#${VERSION:0:7}" >> "${HERE}/.setup/installed"
diff --git a/frameworks/constantpredictor/exec.py b/frameworks/constantpredictor/exec.py
index 6a7aae69c..48b4db7c3 100644
--- a/frameworks/constantpredictor/exec.py
+++ b/frameworks/constantpredictor/exec.py
@@ -1,11 +1,13 @@
 import logging
 
+import pandas as pd
 from sklearn.dummy import DummyClassifier, DummyRegressor
 
 from amlb.benchmark import TaskConfig
 from amlb.data import Dataset
 from amlb.results import save_predictions
 from amlb.utils import Timer, unsparsify
+from frameworks.shared.callee import measure_inference_times
 
 log = logging.getLogger(__name__)
 
@@ -25,9 +27,26 @@ def run(dataset: Dataset, config: TaskConfig):
 
     with Timer() as training:
         predictor.fit(X_train, y_train)
+    log.info(f"Finished fit in {training.duration}s.")
+
     with Timer() as predict:
         predictions = predictor.predict(X_test)
     probabilities = predictor.predict_proba(X_test) if is_classification else None
+    log.info(f"Finished predict in {predict.duration}s.")
+
+    def infer(data):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        return predictor.predict(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files(fmt="parquet"))
+        test_data = X_test if isinstance(X_test, pd.DataFrame) else pd.DataFrame(X_test)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, test_data.sample(1, random_state=i)) for i in range(100)],
+        )
+        log.info(f"Finished inference time measurements.")
 
     save_predictions(dataset=dataset,
                      output_file=config.output_predictions_file,
@@ -39,5 +58,6 @@ def run(dataset: Dataset, config: TaskConfig):
     return dict(
         models_count=1,
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
diff --git a/frameworks/flaml/__init__.py b/frameworks/flaml/__init__.py
index c911edf3b..dcec90325 100644
--- a/frameworks/flaml/__init__.py
+++ b/frameworks/flaml/__init__.py
@@ -17,8 +17,10 @@ def run(dataset, config):
             X=dataset.test.X,
             y=dataset.test.y
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/flaml/exec.py b/frameworks/flaml/exec.py
index 7c450ef47..fd0951a53 100644
--- a/frameworks/flaml/exec.py
+++ b/frameworks/flaml/exec.py
@@ -1,9 +1,12 @@
 import logging
 import os
+from typing import Union
 
+import pandas as pd
 from flaml import AutoML, __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(__name__)
@@ -13,7 +16,6 @@ def run(dataset, config):
     log.info(f"\n**** FLAML [v{__version__}] ****\n")
 
     X_train, y_train = dataset.train.X, dataset.train.y.squeeze()
-    X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
 
     is_classification = config.type == 'classification'
     time_budget = config.max_runtime_seconds
@@ -49,11 +51,31 @@ def run(dataset, config):
                 n_jobs=n_jobs,
                 log_file_name= flaml_log_file_name,
                 time_budget=time_budget, **training_params)
-    
+    log.info(f"Finished fit in {training.duration}s.")
+
+    def infer(data: Union[str, pd.DataFrame]):
+        data = pd.read_parquet(data) if isinstance(data, str) else data
+        predict_fn = aml.predict_proba if is_classification else aml.predict
+        return predict_fn(data)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+        log.info(f"Finished inference time measurements.")
+
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
         predictions = aml.predict(X_test)
     probabilities = aml.predict_proba(X_test) if is_classification else None
-    labels = aml.classes_ if is_classification else None
+    labels = None
+    if is_classification:
+        labels = aml.classes_ if isinstance(aml.classes_, list) else aml.classes_.tolist()
+    log.info(f"Finished predict in {predict.duration}s.")
+
     return result(  
                     output_file=config.output_predictions_file,
                     probabilities=probabilities,
@@ -63,6 +85,7 @@ def run(dataset, config):
                     training_duration=training.duration,
                     predict_duration=predict.duration,
                     probabilities_labels=labels,
+                    inference_times=inference_times,
                 )
 
 
diff --git a/frameworks/flaml/setup.sh b/frameworks/flaml/setup.sh
index 776620157..7fb003789 100755
--- a/frameworks/flaml/setup.sh
+++ b/frameworks/flaml/setup.sh
@@ -3,22 +3,34 @@ HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/microsoft/FLAML.git"}
 PKG=${3:-"flaml"}
+
+
+. ${HERE}/../shared/setup.sh ${HERE} true
+
 if [[ "$VERSION" == "latest" ]]; then
     VERSION="main"
+    OPTIONALS=""
+fi
+
+if [[ "$VERSION" == "benchmark" ]]; then
+    VERSION="stable"
+    OPTIONALS="[catboost]"
+else
+    PIP uninstall -y catboost
 fi
 
-. ${HERE}/../shared/setup.sh ${HERE} true
 
 if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
+    PIP install --no-cache-dir -U "${PKG}${OPTIONALS}<2"
 elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+    PIP install --no-cache-dir -U ${PKG}${OPTIONALS}==${VERSION}
 else
 #    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg=${PKG}
     TARGET_DIR="${HERE}/lib/${PKG}"
     rm -Rf ${TARGET_DIR}
     git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
-    PIP install -U -e ${TARGET_DIR}
+    PIP install -U -e ${TARGET_DIR}${OPTIONALS}
 fi
 
 PY -c "from flaml import __version__; print(__version__)" >> "${HERE}/.setup/installed"
+echo ${OPTIONALS} >> "${HERE}/.setup/installed"
diff --git a/frameworks/hyperoptsklearn/setup.sh b/frameworks/hyperoptsklearn/setup.sh
index c574f1848..b2cc19005 100755
--- a/frameworks/hyperoptsklearn/setup.sh
+++ b/frameworks/hyperoptsklearn/setup.sh
@@ -3,21 +3,27 @@ HERE=$(dirname "$0")
 VERSION=${1:-"stable"}
 REPO=${2:-"https://github.com/hyperopt/hyperopt-sklearn.git"}
 PKG=${3:-"hyperopt-sklearn"}
-if [[ "$VERSION" == "latest" ]]; then
+
+if [[ "$VERSION" == "latest" || "$VERSION" == "stable" ]]; then
     VERSION="master"
 fi
 
 . ${HERE}/../shared/setup.sh ${HERE} true
 
-if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U ${PKG}
-elif [[ "$VERSION" =~ ^[0-9] ]]; then
-    PIP install --no-cache-dir -U ${PKG}==${VERSION}
+if [[ "$VERSION" =~ ^[0-9] ]]; then
+    PIP install git+https://github.com/hyperopt/hyperopt-sklearn@${VERSION}
 else
 #    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg=${PKG}
     LIB=$(echo ${PKG} | sed "s/\[.*\]//")
     TARGET_DIR="${HERE}/lib/${LIB}"
     rm -Rf ${TARGET_DIR}
-    git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
+    if [[ "$VERSION" =~ ^# ]]; then
+        git clone --recurse-submodules --shallow-submodules ${REPO} ${TARGET_DIR}
+        cd ${TARGET_DIR}
+        git checkout "${VERSION:1}"
+        cd ${HERE}
+    else
+        git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
+    fi
     PIP install -U -e ${HERE}/lib/${PKG}
 fi
diff --git a/frameworks/lightautoml/__init__.py b/frameworks/lightautoml/__init__.py
index 97c09fa0e..4c9654850 100644
--- a/frameworks/lightautoml/__init__.py
+++ b/frameworks/lightautoml/__init__.py
@@ -21,8 +21,10 @@ def run(dataset: Dataset, config: TaskConfig):
         target=dict(
             name=dataset.target.name,
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/lightautoml/exec.py b/frameworks/lightautoml/exec.py
index aee255902..4a18052bb 100644
--- a/frameworks/lightautoml/exec.py
+++ b/frameworks/lightautoml/exec.py
@@ -2,16 +2,20 @@
 import os
 import pickle
 import warnings
+from typing import Union
 
 import matplotlib
 import numpy as np
+import pandas as pd
+
 matplotlib.use("agg")  # no need for tk
 
 from lightautoml.tasks import Task
 from lightautoml.automl.presets.tabular_presets import TabularAutoML, TabularUtilizedAutoML
 from lightautoml import __version__
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(__name__)
@@ -36,10 +40,25 @@ def run(dataset, config):
     log.info("Training...")
     with Timer() as training:
         automl.fit_predict(train_data=df_train, roles={'target': label})
+    log.info(f"Finished fit in {training.duration}s.")
+
+    def infer(data: Union[str, pd.DataFrame]):
+        batch = pd.read_parquet(data) if isinstance(data, str) else data
+        return automl.predict(batch)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+    log.info(f"Finished inference time measurements.")
+
 
-    X_test, y_test = dataset.test.X, dataset.test.y
     log.info("Predicting on the test set...")
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y
         preds = automl.predict(X_test).data
 
     probabilities_labels = None
@@ -65,6 +84,8 @@ def run(dataset, config):
 
     log.debug(probabilities)
     log.debug(config.output_predictions_file)
+    log.info(f"Finished predict in {predict.duration}s.")
+
 
     save_artifacts(automl, config)
 
@@ -75,6 +96,7 @@ def run(dataset, config):
         predictions=predictions,
         training_duration=training.duration,
         predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/mljarsupervised/__init__.py b/frameworks/mljarsupervised/__init__.py
index 9bee9f4a5..b15d780f8 100644
--- a/frameworks/mljarsupervised/__init__.py
+++ b/frameworks/mljarsupervised/__init__.py
@@ -19,8 +19,10 @@ def run(dataset: Dataset, config: TaskConfig):
             X=dataset.test.X,
             y=dataset.test.y
         ),
-        problem_type=dataset.type.name
+        problem_type=dataset.type.name,
     )
+    if config.measure_inference_time:
+        data["inference_subsample_files"] = dataset.inference_subsample_files(fmt="parquet")
     options = dict(
         serialization=dict(sparse_dataframe_deserialized_format='dense')
     )
diff --git a/frameworks/mljarsupervised/exec.py b/frameworks/mljarsupervised/exec.py
index 653d9cfd6..64d3fee4a 100644
--- a/frameworks/mljarsupervised/exec.py
+++ b/frameworks/mljarsupervised/exec.py
@@ -1,15 +1,19 @@
 import os
 import shutil
 import logging
+from typing import Union
 
 import numpy as np
 import matplotlib
+import pandas as pd
+
 matplotlib.use("agg")  # no need for tk
 
 import supervised
 from supervised.automl import AutoML
 
-from frameworks.shared.callee import call_run, result, output_subdir
+from frameworks.shared.callee import call_run, result, output_subdir, \
+    measure_inference_times
 from frameworks.shared.utils import Timer
 
 log = logging.getLogger(os.path.basename(__file__))
@@ -42,7 +46,6 @@ def run(dataset, config):
     }
 
     X_train, y_train = dataset.train.X, dataset.train.y.squeeze()
-    X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
 
     automl = AutoML(
         results_path=results_path,
@@ -55,8 +58,24 @@ def run(dataset, config):
 
     with Timer() as training:
         automl.fit(X_train, y_train)
+    log.info(f"Finished fit in {training.duration}s.")
+
+
+    def infer(data: Union[str, pd.DataFrame]):
+        batch = pd.read_parquet(data) if isinstance(data, str) else data
+        return automl.predict_all(batch)
+
+    inference_times = {}
+    if config.measure_inference_time:
+        inference_times["file"] = measure_inference_times(infer, dataset.inference_subsample_files)
+        inference_times["df"] = measure_inference_times(
+            infer,
+            [(1, dataset.test.X.sample(1, random_state=i)) for i in range(100)],
+        )
+    log.info(f"Finished inference time measurements.")
 
     with Timer() as predict:
+        X_test, y_test = dataset.test.X, dataset.test.y.squeeze()
         preds = automl.predict_all(X_test)
 
     predictions, probabilities, probabilities_labels = None, None, None
@@ -75,6 +94,7 @@ def run(dataset, config):
         probabilities = preds[probabilities_labels].values
     else:
         predictions = preds["prediction"].values
+    log.info(f"Finished predict in {predict.duration}s.")
 
     # clean the results
     if not config.framework_params.get("_save_artifacts", False):
@@ -88,7 +108,8 @@ def run(dataset, config):
         probabilities_labels=probabilities_labels,
         models_count=len(automl._models),
         training_duration=training.duration,
-        predict_duration=predict.duration
+        predict_duration=predict.duration,
+        inference_times=inference_times,
     )
 
 
diff --git a/frameworks/mlr3automl/setup.sh b/frameworks/mlr3automl/setup.sh
index cd5225c04..b939770e4 100755
--- a/frameworks/mlr3automl/setup.sh
+++ b/frameworks/mlr3automl/setup.sh
@@ -24,7 +24,7 @@ fi
 
 . $HERE/../shared/setup.sh "$HERE"
 if [[ -x "$(command -v apt-get)" ]]; then
-SUDO apt-get update
+  SUDO apt-get update
   SUDO apt-get install -y software-properties-common dirmngr
   SUDO apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
   SUDO add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
diff --git a/frameworks/oboe/exec.py b/frameworks/oboe/exec.py
index 37feebe06..bed388dc7 100644
--- a/frameworks/oboe/exec.py
+++ b/frameworks/oboe/exec.py
@@ -2,8 +2,11 @@
 import os
 import sys
 
+from sklearn.model_selection import StratifiedKFold
+import numpy as np
+
 sys.path.append("{}/lib/oboe/automl".format(os.path.realpath(os.path.dirname(__file__))))
-from auto_learner import AutoLearner
+from oboe import AutoLearner
 
 from frameworks.shared.callee import call_run, result
 from frameworks.shared.utils import Timer
@@ -11,9 +14,46 @@
 log = logging.getLogger(__name__)
 
 
+def kfold_fit_validate(self, x_train, y_train, n_folds, random_state=None):
+    """Performs k-fold cross validation on a training dataset. Note that this is the function used to fill entries
+    of the error matrix.
+    Args:
+        x_train (np.ndarray): Features of the training dataset.
+        y_train (np.ndarray): Labels of the training dataset.
+        n_folds (int):        Number of folds to use for cross validation.
+    Returns:
+        float: Mean of k-fold cross validation error.
+        np.ndarray: Predictions on the training dataset from cross validation.
+    """
+    y_predicted = np.empty(y_train.shape)
+    cv_errors = np.empty(n_folds)
+    kf = StratifiedKFold(n_folds, shuffle=True, random_state=random_state)
+    for i, (train_idx, test_idx) in enumerate(kf.split(x_train, y_train)):
+        x_tr = x_train[train_idx, :]
+        y_tr = y_train[train_idx]
+        x_te = x_train[test_idx, :]
+        y_te = y_train[test_idx]
+        model = self.instantiate()
+        if len(np.unique(y_tr)) > 1:
+            model.fit(x_tr, y_tr)
+            y_predicted[test_idx] = np.expand_dims(model.predict(x_te), axis=1)
+        else:
+            y_predicted[test_idx] = y_tr[0]
+        cv_errors[i] = self.error(y_te, y_predicted[test_idx])
+    self.cv_error = cv_errors.mean()
+    self.cv_predictions = y_predicted
+    self.sampled = True
+    if self.verbose:
+        print("{} {} complete.".format(self.algorithm, self.hyperparameters))
+    return cv_errors, y_predicted
+
+
 def run(dataset, config):
-    log.info(f"\n**** Oboe [{config.framework_version}] ****\n")
+    log.info(f"\n**** Applying monkey patch ****\n")
+    from oboe.model import Model
+    Model.kfold_fit_validate = kfold_fit_validate
 
+    log.info(f"\n**** Oboe [{config.framework_version}] ****\n")
     is_classification = config.type == 'classification'
     if not is_classification:
         # regression currently fails (as of 26.02.2019: still under development state by oboe team)
@@ -32,6 +72,7 @@ def run(dataset, config):
                       n_cores=n_cores,
                       runtime_limit=config.max_runtime_seconds,
                       **training_params)
+    aml.error_matrix = aml.error_matrix.to_numpy()
 
     aml_models = lambda: [aml.ensemble, *aml.ensemble.base_learners] if len(aml.ensemble.base_learners) > 0 else []
 
diff --git a/frameworks/oboe/requirements.txt b/frameworks/oboe/requirements.txt
deleted file mode 100644
index a489529ca..000000000
--- a/frameworks/oboe/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-cvxpy>=1.0,<2.0
-numpy==1.16.4
-scipy==1.4.1
-pandas==0.24.2
-scikit-learn==0.22.1
-multiprocess>=0.70.5
-openml==0.10.2
-mkl>=1.0.0
-tensorly
diff --git a/frameworks/oboe/setup.sh b/frameworks/oboe/setup.sh
index 212bf9ffa..016d07241 100755
--- a/frameworks/oboe/setup.sh
+++ b/frameworks/oboe/setup.sh
@@ -21,5 +21,5 @@ else
     PIP install -U -e ${TARGET_DIR}
 fi
 
-cat ${HERE}/requirements.txt | sed '/^$/d' | while read -r i; do PIP install --no-cache-dir -U "$i"; done
+#cat ${HERE}/requirements.txt | sed '/^$/d' | while read -r i; do PIP install --no-cache-dir -U "$i"; done
 #PIP install --no-cache-dir -U -e git+https://github.com/udellgroup/oboe.git@${VERSION}#egg=oboe
diff --git a/frameworks/ranger/exec.R b/frameworks/ranger/exec.R
index a998badeb..5259cfed0 100644
--- a/frameworks/ranger/exec.R
+++ b/frameworks/ranger/exec.R
@@ -11,16 +11,17 @@ run <- function(train_file, test_file, output_predictions_file, cores=1, meta_re
   colnames(train) <- make.names(colnames(train))
   colnames(test) <- make.names(colnames(test))
   target <- colnames(train)[ncol(train)]
-  is_classifaction <- task_type == "classification"
-  task_class <- if (is_classifaction) TaskClassif else TaskRegr
+  is_classification <- task_type == "classification"
+  task_class <- if (is_classification) TaskClassif else TaskRegr
   train_task <- task_class$new(id="ranger.benchmark.train", backend = train, target = target)
   test_task <- task_class$new(id="ranger.benchmark.test", backend = test, target = target)
 
   learner <- po("removeconstants") %>>%
          po("imputeoor") %>>%
-         po("learner", learner = lrn(if (is_classifaction) "classif.ranger" else "regr.ranger",
-                                     num.threads = cores,
-                                     predict_type = if (is_classifaction) "prob" else "response"))
+         po("learner",
+            learner = lrn(if (is_classification) "classif.ranger" else "regr.ranger",
+                          num.threads = cores,
+                          predict_type = if (is_classification) "prob" else "response"))
 
   mod <- NULL
   preds <- NULL
@@ -28,7 +29,7 @@ run <- function(train_file, test_file, output_predictions_file, cores=1, meta_re
   prediction <- function() preds <<- learner$predict(test_task)[[mod]]
   train_duration <- system.time(training())[['elapsed']]
   predict_duration <- system.time(prediction())[['elapsed']]
-  if (is_classifaction) {
+  if (is_classification) {
     labels <- colnames(preds$prob)
     as_label <- function(x) labels[[x]]
     predictions <- cbind(preds$prob,
diff --git a/frameworks/ranger/exec.py b/frameworks/ranger/exec.py
index 3fb6905de..1ad60705e 100644
--- a/frameworks/ranger/exec.py
+++ b/frameworks/ranger/exec.py
@@ -14,11 +14,12 @@ def run(dataset: Dataset, config: TaskConfig):
     log.info("\n**** Random Forest (R) ****\n")
 
     here = dir_of(__file__)
-    meta_results_file = os.path.join(config.output_dir, "meta_results.csv")
+    meta_results_file = os.path.join(os.path.dirname(config.output_metadata_file), "meta_results.csv")
     run_cmd(("Rscript --vanilla -e \""
-             "source('{script}'); "
+             ".libPaths('{package_directory}'); source('{script}'); "
              "run('{train}', '{test}', '{output}', cores={cores}, meta_results_file='{meta_results}', task_type='{task_type}')"
              "\"").format(
+        package_directory=os.path.join(here, "lib"),
         script=os.path.join(here, 'exec.R'),
         train=dataset.train.path,
         test=dataset.test.path,
diff --git a/frameworks/ranger/setup.sh b/frameworks/ranger/setup.sh
index f279b3ac5..d36837dc1 100755
--- a/frameworks/ranger/setup.sh
+++ b/frameworks/ranger/setup.sh
@@ -15,7 +15,8 @@ if [[ -x "$(command -v apt-get)" ]]; then
     SUDO apt-get install -y r-base r-base-dev
 fi
 #PIP install --no-cache-dir -r $HERE/requirements.txt
+LIB="${HERE}/lib/"
+mkdir "${LIB}"
 
-Rscript -e 'options(install.packages.check.source="no"); install.packages(c("ranger", "mlr3", "mlr3learners", "mlr3pipelines", "farff"), repos="https://cloud.r-project.org/")'
-
+Rscript -e 'options(install.packages.check.source="no"); install.packages(c("ranger", "mlr3", "mlr3learners", "mlr3pipelines", "farff"), repos="https://cloud.r-project.org/", lib="'"${LIB}"'")'
 Rscript -e 'packageVersion("ranger")' | awk '{print $2}' | sed "s/[‘’]//g" >> "${HERE}/.setup/installed"
diff --git a/frameworks/shared/callee.py b/frameworks/shared/callee.py
index 3bf70dd3c..84fe1c6dc 100644
--- a/frameworks/shared/callee.py
+++ b/frameworks/shared/callee.py
@@ -1,11 +1,15 @@
 import logging
 import os
+import pathlib
 import re
 import signal
 import sys
+from collections import defaultdict
+from typing import Callable, Any, Tuple, Union, TypeVar, TYPE_CHECKING
 
 from .utils import InterruptTimeout, Namespace as ns, json_dump, json_loads, kill_proc_tree, touch
-from .utils import deserialize_data, serialize_data
+from .utils import deserialize_data, serialize_data, Timer
+
 
 log = logging.getLogger(__name__)
 
@@ -17,6 +21,7 @@ class FrameworkError(Exception):
 def result(output_file=None,
            predictions=None, truth=None,
            probabilities=None, probabilities_labels=None,
+           optional_columns=None,
            target_is_encoded=False,
            error_message=None,
            models_count=None,
@@ -69,7 +74,7 @@ def load_data(name, path, **_):
                               wait_retry_secs=10):
             result = run_fn(ds, config)
             res = dict(result)
-            for name in ['predictions', 'truth', 'probabilities']:
+            for name in ['predictions', 'truth', 'probabilities', 'optional_columns']:
                 arr = result[name]
                 if arr is not None:
                     path = os.path.join(config.result_dir, '.'.join([name, 'data']))
@@ -84,4 +89,25 @@ def load_data(name, path, **_):
         # ensure there's no subprocess left
         kill_proc_tree(include_parent=False, timeout=5)
 
+    inference_measurements = res.get("others", {}).get("inference_times")
+    if inference_measurements:
+        inference_file = pathlib.Path(config.result_file).parent / "inference_times.json"
+        json_dump(inference_measurements, inference_file, style="compact")
+        res["others"]["inference_times"] = str(inference_file)
     json_dump(res, config.result_file, style='compact')
+
+try:
+    import pandas as pd
+    DATA_TYPES = Union[str, pd.DataFrame]
+except ImportError:
+    DATA_TYPES = str
+
+DATA_INPUT = TypeVar("DATA_INPUT", bound=DATA_TYPES)
+
+def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:
+    inference_times = defaultdict(list)
+    for subsample_size, subsample_path in files:
+        with Timer() as predict:
+            predict_fn(subsample_path)
+        inference_times[subsample_size].append(predict.duration)
+    return inference_times
\ No newline at end of file
diff --git a/frameworks/shared/caller.py b/frameworks/shared/caller.py
index b4c9b0210..8422b98d4 100644
--- a/frameworks/shared/caller.py
+++ b/frameworks/shared/caller.py
@@ -1,9 +1,10 @@
 import gc
 import logging
 import os
+import pathlib
 import re
 from tempfile import TemporaryDirectory, mktemp
-from typing import Optional, Union
+from typing import List, Optional, Union
 
 import numpy as np
 
@@ -11,6 +12,7 @@
 from amlb.data import Dataset
 from amlb.resources import config as rconfig
 from amlb.results import NoResultError, save_predictions
+from amlb.utils import json_dump, Namespace
 
 from .utils import Namespace as ns, Timer, dir_of, run_cmd, json_dumps, json_load, profile
 from .utils import is_serializable_data, deserialize_data, serialize_data
@@ -82,7 +84,8 @@ def run_in_venv(caller_file, script_file: str, *args,
                 input_data: Union[dict, ns], dataset: Dataset, config: TaskConfig,
                 options: Union[None, dict, ns] = None,
                 process_results=None,
-                python_exec=None):
+                python_exec=None,
+                retained_env_vars: Optional[List[str]] = ['TMP', 'TEMP', 'TMPDIR']):
     here = dir_of(caller_file)
     if python_exec is None:  # use local virtual env by default
         python_exec = venv_python_exec(here)
@@ -93,6 +96,13 @@ def run_in_venv(caller_file, script_file: str, *args,
     ser_config = options['serialization']
     env = options['env'] or ns()
 
+    # Add any env variables specified if they are defined in the environment
+    if retained_env_vars:
+        for env_var in retained_env_vars:
+            env_val = os.environ.get(env_var)
+            if env_val is not None:
+                env[env_var] = env_val
+
     with TemporaryDirectory(prefix='amlb_', suffix='_xproc') as tmpdir:
 
         ds = _make_input_dataset(input_data, dataset, tmpdir, serialization=ser_config)
@@ -115,7 +125,7 @@ def run_in_venv(caller_file, script_file: str, *args,
             PYTHONPATH=os.pathsep.join([
                 rconfig().root_dir,
             ]),
-            AMLB_PATH=os.path.join(rconfig().root_dir, "amlb"),
+            AMLB_PATH=os.path.join(rconfig().root_dir),
             AMLB_LOG_TRACE=str(logging.TRACE if hasattr(logging, 'TRACE') else ''),
             **{k: str(v) for k, v in env}
         )
@@ -141,9 +151,16 @@ def run_in_venv(caller_file, script_file: str, *args,
         if res.error_message is not None:
             raise NoResultError(res.error_message)
 
-        for name in ['predictions', 'truth', 'probabilities']:
+        for name in ['predictions', 'truth', 'probabilities', 'optional_columns']:
             res[name] = deserialize_data(res[name], config=ser_config) if res[name] is not None else None
 
+        inference_filepath = Namespace.dict(res.others).get("inference_times")
+        if inference_filepath:
+            inference_times = json_load(inference_filepath)
+            inference_filepath = pathlib.Path(res.output_file).parent / "inference.json"
+            json_dump(inference_times, inference_filepath)
+            res["others"]["inference_times"] = inference_times
+
         if callable(process_results):
             res = process_results(res)
 
@@ -156,6 +173,7 @@ def run_in_venv(caller_file, script_file: str, *args,
                                     else dataset.test.y),
                              probabilities=res.probabilities,
                              probabilities_labels=res.probabilities_labels,
+                             optional_columns=res.optional_columns,
                              target_is_encoded=res.target_is_encoded)
 
         return dict(
diff --git a/frameworks/shared/requirements.in b/frameworks/shared/requirements.in
index b09b57f51..bfe9e1399 100644
--- a/frameworks/shared/requirements.in
+++ b/frameworks/shared/requirements.in
@@ -1,5 +1,4 @@
 psutil>=5.4
-ruamel.yaml>=0.15
 
 pyarrow>=4.0
-#tables>=3.6    # if using hdf serializer (let the client choose)
+# tables>=3.6    # if using hdf serializer (let the client choose)
diff --git a/frameworks/shared/requirements.txt b/frameworks/shared/requirements.txt
index 53569f1cb..4f291b10b 100644
--- a/frameworks/shared/requirements.txt
+++ b/frameworks/shared/requirements.txt
@@ -1,16 +1,11 @@
 #
-# This file is autogenerated by pip-compile
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
 #
-#    pip-compile frameworks/shared/requirements.in
+#    pip-compile --output-file=frameworks/shared/requirements.txt frameworks/shared/requirements.in
 #
-numpy==1.21.0
+numpy==1.24.2
     # via pyarrow
 psutil==5.8.0
     # via -r frameworks/shared/requirements.in
-pyarrow==4.0.1
-    # via -r frameworks/shared/requirements.in
-ruamel.yaml.clib==0.2.2
-    # via ruamel.yaml
-ruamel.yaml==0.17.4
-    # via -r frameworks/shared/requirements.in
+pyarrow==11.0.0
diff --git a/frameworks/shared/utils.py b/frameworks/shared/utils.py
index 23f2b6555..abcff3717 100644
--- a/frameworks/shared/utils.py
+++ b/frameworks/shared/utils.py
@@ -3,7 +3,6 @@
 import logging
 import os
 import sys
-from types import ModuleType
 
 
 def setup_logger():
@@ -20,6 +19,8 @@ def setup_logger():
 
 setup_logger()
 
+__no_export = set(dir())  # all variables defined above this are not exported
+
 
 def load_module(name, path):
     spec = importlib.util.spec_from_file_location(name, path)
@@ -29,14 +30,20 @@ def load_module(name, path):
     return module
 
 
-def load_utils():
-    amlb_path = os.environ.get("AMLB_PATH")
+def load_amlb_module(mod, amlb_path=None):
+    amlb_path = amlb_path or os.environ.get("AMLB_PATH")
     if amlb_path:
-        return load_module("amlb.utils", os.path.join(amlb_path, "utils", "__init__.py"))
-    return import_module("amlb.utils")
+        tokens = mod.split('.')
+        mod_path = os.path.join(amlb_path, *tokens)
+        if os.path.isdir(mod_path):
+            tokens.append("__init__.py")
+            mod_path = os.path.join(amlb_path, *tokens)
+        return load_module(mod, mod_path)
+    return import_module(mod)
 
 
-utils = load_utils()
+utils = load_amlb_module("amlb.utils")
 # unorthodox for it's only now that we can safely import those functions
 from amlb.utils import *
 
+__all__ = [s for s in dir() if not s.startswith('_') and s not in __no_export]
diff --git a/reports/datasets/1h/metadata.csv b/reports/old/datasets/1h/metadata.csv
similarity index 100%
rename from reports/datasets/1h/metadata.csv
rename to reports/old/datasets/1h/metadata.csv
diff --git a/reports/datasets/4h/metadata.csv b/reports/old/datasets/4h/metadata.csv
similarity index 100%
rename from reports/datasets/4h/metadata.csv
rename to reports/old/datasets/4h/metadata.csv
diff --git a/reports/duration_comparison.ipynb b/reports/old/duration_comparison.ipynb
similarity index 100%
rename from reports/duration_comparison.ipynb
rename to reports/old/duration_comparison.ipynb
diff --git a/reports/graphics/1h/binary_norm_score_heat.png b/reports/old/graphics/1h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_heat.png
rename to reports/old/graphics/1h/binary_norm_score_heat.png
diff --git a/reports/graphics/1h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/1h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/1h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/1h/binary_norm_score_stripplot.png b/reports/old/graphics/1h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/1h/binary_norm_score_stripplot.png
rename to reports/old/graphics/1h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/1h/binary_results_stripplot.png b/reports/old/graphics/1h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/1h/binary_results_stripplot.png
rename to reports/old/graphics/1h/binary_results_stripplot.png
diff --git a/reports/graphics/1h/binary_score_heat.png b/reports/old/graphics/1h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/1h/binary_score_heat.png
rename to reports/old/graphics/1h/binary_score_heat.png
diff --git a/reports/graphics/1h/binary_score_parallel_ccord.png b/reports/old/graphics/1h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/binary_score_parallel_ccord.png
rename to reports/old/graphics/1h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/1h/multiclass_norm_score_heat.png b/reports/old/graphics/1h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_heat.png
rename to reports/old/graphics/1h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/1h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/1h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/1h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/1h/multiclass_norm_score_stripplot.png b/reports/old/graphics/1h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/1h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/1h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/1h/multiclass_results_stripplot.png b/reports/old/graphics/1h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/1h/multiclass_results_stripplot.png
rename to reports/old/graphics/1h/multiclass_results_stripplot.png
diff --git a/reports/graphics/1h/multiclass_score_heat.png b/reports/old/graphics/1h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/1h/multiclass_score_heat.png
rename to reports/old/graphics/1h/multiclass_score_heat.png
diff --git a/reports/graphics/1h/multiclass_score_parallel_ccord.png b/reports/old/graphics/1h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/1h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/1h/multiclass_score_parallel_ccord.png
diff --git a/reports/graphics/4h/binary_norm_score_heat.png b/reports/old/graphics/4h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_heat.png
rename to reports/old/graphics/4h/binary_norm_score_heat.png
diff --git a/reports/graphics/4h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/4h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/4h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/4h/binary_norm_score_stripplot.png b/reports/old/graphics/4h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/4h/binary_norm_score_stripplot.png
rename to reports/old/graphics/4h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/4h/binary_results_stripplot.png b/reports/old/graphics/4h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/4h/binary_results_stripplot.png
rename to reports/old/graphics/4h/binary_results_stripplot.png
diff --git a/reports/graphics/4h/binary_score_heat.png b/reports/old/graphics/4h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/4h/binary_score_heat.png
rename to reports/old/graphics/4h/binary_score_heat.png
diff --git a/reports/graphics/4h/binary_score_parallel_ccord.png b/reports/old/graphics/4h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/binary_score_parallel_ccord.png
rename to reports/old/graphics/4h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/4h/multiclass_norm_score_heat.png b/reports/old/graphics/4h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_heat.png
rename to reports/old/graphics/4h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/4h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/4h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/4h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/4h/multiclass_norm_score_stripplot.png b/reports/old/graphics/4h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/4h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/4h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/4h/multiclass_results_stripplot.png b/reports/old/graphics/4h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/4h/multiclass_results_stripplot.png
rename to reports/old/graphics/4h/multiclass_results_stripplot.png
diff --git a/reports/graphics/4h/multiclass_score_heat.png b/reports/old/graphics/4h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/4h/multiclass_score_heat.png
rename to reports/old/graphics/4h/multiclass_score_heat.png
diff --git a/reports/graphics/4h/multiclass_score_parallel_ccord.png b/reports/old/graphics/4h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/4h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/4h/multiclass_score_parallel_ccord.png
diff --git a/reports/graphics/8h/binary_norm_score_heat.png b/reports/old/graphics/8h/binary_norm_score_heat.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_heat.png
rename to reports/old/graphics/8h/binary_norm_score_heat.png
diff --git a/reports/graphics/8h/binary_norm_score_parallel_ccord.png b/reports/old/graphics/8h/binary_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_parallel_ccord.png
rename to reports/old/graphics/8h/binary_norm_score_parallel_ccord.png
diff --git a/reports/graphics/8h/binary_norm_score_stripplot.png b/reports/old/graphics/8h/binary_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/8h/binary_norm_score_stripplot.png
rename to reports/old/graphics/8h/binary_norm_score_stripplot.png
diff --git a/reports/graphics/8h/binary_results_stripplot.png b/reports/old/graphics/8h/binary_results_stripplot.png
similarity index 100%
rename from reports/graphics/8h/binary_results_stripplot.png
rename to reports/old/graphics/8h/binary_results_stripplot.png
diff --git a/reports/graphics/8h/binary_score_heat.png b/reports/old/graphics/8h/binary_score_heat.png
similarity index 100%
rename from reports/graphics/8h/binary_score_heat.png
rename to reports/old/graphics/8h/binary_score_heat.png
diff --git a/reports/graphics/8h/binary_score_parallel_ccord.png b/reports/old/graphics/8h/binary_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/binary_score_parallel_ccord.png
rename to reports/old/graphics/8h/binary_score_parallel_ccord.png
diff --git a/reports/graphics/8h/multiclass_norm_score_heat.png b/reports/old/graphics/8h/multiclass_norm_score_heat.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_heat.png
rename to reports/old/graphics/8h/multiclass_norm_score_heat.png
diff --git a/reports/graphics/8h/multiclass_norm_score_parallel_ccord.png b/reports/old/graphics/8h/multiclass_norm_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_parallel_ccord.png
rename to reports/old/graphics/8h/multiclass_norm_score_parallel_ccord.png
diff --git a/reports/graphics/8h/multiclass_norm_score_stripplot.png b/reports/old/graphics/8h/multiclass_norm_score_stripplot.png
similarity index 100%
rename from reports/graphics/8h/multiclass_norm_score_stripplot.png
rename to reports/old/graphics/8h/multiclass_norm_score_stripplot.png
diff --git a/reports/graphics/8h/multiclass_results_stripplot.png b/reports/old/graphics/8h/multiclass_results_stripplot.png
similarity index 100%
rename from reports/graphics/8h/multiclass_results_stripplot.png
rename to reports/old/graphics/8h/multiclass_results_stripplot.png
diff --git a/reports/graphics/8h/multiclass_score_heat.png b/reports/old/graphics/8h/multiclass_score_heat.png
similarity index 100%
rename from reports/graphics/8h/multiclass_score_heat.png
rename to reports/old/graphics/8h/multiclass_score_heat.png
diff --git a/reports/graphics/8h/multiclass_score_parallel_ccord.png b/reports/old/graphics/8h/multiclass_score_parallel_ccord.png
similarity index 100%
rename from reports/graphics/8h/multiclass_score_parallel_ccord.png
rename to reports/old/graphics/8h/multiclass_score_parallel_ccord.png
diff --git a/reports/input/results_large-8c4h.csv b/reports/old/input/results_large-8c4h.csv
similarity index 100%
rename from reports/input/results_large-8c4h.csv
rename to reports/old/input/results_large-8c4h.csv
diff --git a/reports/input/results_large-8c4h_ref.csv b/reports/old/input/results_large-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_large-8c4h_ref.csv
rename to reports/old/input/results_large-8c4h_ref.csv
diff --git a/reports/input/results_large-8c8h.csv b/reports/old/input/results_large-8c8h.csv
similarity index 100%
rename from reports/input/results_large-8c8h.csv
rename to reports/old/input/results_large-8c8h.csv
diff --git a/reports/input/results_large-8c8h_ref.csv b/reports/old/input/results_large-8c8h_ref.csv
similarity index 100%
rename from reports/input/results_large-8c8h_ref.csv
rename to reports/old/input/results_large-8c8h_ref.csv
diff --git a/reports/input/results_medium-4c1h.csv b/reports/old/input/results_medium-4c1h.csv
similarity index 100%
rename from reports/input/results_medium-4c1h.csv
rename to reports/old/input/results_medium-4c1h.csv
diff --git a/reports/input/results_medium-4c1h_ref.csv b/reports/old/input/results_medium-4c1h_ref.csv
similarity index 100%
rename from reports/input/results_medium-4c1h_ref.csv
rename to reports/old/input/results_medium-4c1h_ref.csv
diff --git a/reports/input/results_medium-4c4h.csv b/reports/old/input/results_medium-4c4h.csv
similarity index 100%
rename from reports/input/results_medium-4c4h.csv
rename to reports/old/input/results_medium-4c4h.csv
diff --git a/reports/input/results_medium-4c4h_ref.csv b/reports/old/input/results_medium-4c4h_ref.csv
similarity index 100%
rename from reports/input/results_medium-4c4h_ref.csv
rename to reports/old/input/results_medium-4c4h_ref.csv
diff --git a/reports/input/results_medium-8c1h.csv b/reports/old/input/results_medium-8c1h.csv
similarity index 100%
rename from reports/input/results_medium-8c1h.csv
rename to reports/old/input/results_medium-8c1h.csv
diff --git a/reports/input/results_medium-8c1h_ref.csv b/reports/old/input/results_medium-8c1h_ref.csv
similarity index 100%
rename from reports/input/results_medium-8c1h_ref.csv
rename to reports/old/input/results_medium-8c1h_ref.csv
diff --git a/reports/input/results_medium-8c4h.csv b/reports/old/input/results_medium-8c4h.csv
similarity index 100%
rename from reports/input/results_medium-8c4h.csv
rename to reports/old/input/results_medium-8c4h.csv
diff --git a/reports/input/results_medium-8c4h_ref.csv b/reports/old/input/results_medium-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_medium-8c4h_ref.csv
rename to reports/old/input/results_medium-8c4h_ref.csv
diff --git a/reports/input/results_old.csv b/reports/old/input/results_old.csv
similarity index 100%
rename from reports/input/results_old.csv
rename to reports/old/input/results_old.csv
diff --git a/reports/input/results_small-2c1h.csv b/reports/old/input/results_small-2c1h.csv
similarity index 100%
rename from reports/input/results_small-2c1h.csv
rename to reports/old/input/results_small-2c1h.csv
diff --git a/reports/input/results_small-2c1h_ref.csv b/reports/old/input/results_small-2c1h_ref.csv
similarity index 100%
rename from reports/input/results_small-2c1h_ref.csv
rename to reports/old/input/results_small-2c1h_ref.csv
diff --git a/reports/input/results_small-8c1h.csv b/reports/old/input/results_small-8c1h.csv
similarity index 100%
rename from reports/input/results_small-8c1h.csv
rename to reports/old/input/results_small-8c1h.csv
diff --git a/reports/input/results_small-8c1h_ref.csv b/reports/old/input/results_small-8c1h_ref.csv
similarity index 100%
rename from reports/input/results_small-8c1h_ref.csv
rename to reports/old/input/results_small-8c1h_ref.csv
diff --git a/reports/input/results_small-8c4h.csv b/reports/old/input/results_small-8c4h.csv
similarity index 100%
rename from reports/input/results_small-8c4h.csv
rename to reports/old/input/results_small-8c4h.csv
diff --git a/reports/input/results_small-8c4h_ref.csv b/reports/old/input/results_small-8c4h_ref.csv
similarity index 100%
rename from reports/input/results_small-8c4h_ref.csv
rename to reports/old/input/results_small-8c4h_ref.csv
diff --git a/reports/input/results_valid.csv b/reports/old/input/results_valid.csv
similarity index 100%
rename from reports/input/results_valid.csv
rename to reports/old/input/results_valid.csv
diff --git a/reports/input/results_valid_ref.csv b/reports/old/input/results_valid_ref.csv
similarity index 100%
rename from reports/input/results_valid_ref.csv
rename to reports/old/input/results_valid_ref.csv
diff --git a/reports/old/readme.md b/reports/old/readme.md
new file mode 100644
index 000000000..5d6ae51d1
--- /dev/null
+++ b/reports/old/readme.md
@@ -0,0 +1,3 @@
+# Reports for 2019 AutoML Benchmark
+
+These notebooks are built based on a previous iteration of the AutoML benchmark output, so will not work out-of-the-box with the new result files.
diff --git a/reports/reports.ipynb b/reports/old/reports.ipynb
similarity index 100%
rename from reports/reports.ipynb
rename to reports/old/reports.ipynb
diff --git a/reports/tables/1h/all_results.csv b/reports/old/tables/1h/all_results.csv
similarity index 100%
rename from reports/tables/1h/all_results.csv
rename to reports/old/tables/1h/all_results.csv
diff --git a/reports/tables/1h/folds_leaderboard.csv b/reports/old/tables/1h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/1h/folds_leaderboard.csv
rename to reports/old/tables/1h/folds_leaderboard.csv
diff --git a/reports/tables/1h/models_summary.csv b/reports/old/tables/1h/models_summary.csv
similarity index 100%
rename from reports/tables/1h/models_summary.csv
rename to reports/old/tables/1h/models_summary.csv
diff --git a/reports/tables/1h/norm_score_summary.csv b/reports/old/tables/1h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/1h/norm_score_summary.csv
rename to reports/old/tables/1h/norm_score_summary.csv
diff --git a/reports/tables/1h/result_summary.csv b/reports/old/tables/1h/result_summary.csv
similarity index 100%
rename from reports/tables/1h/result_summary.csv
rename to reports/old/tables/1h/result_summary.csv
diff --git a/reports/tables/1h/score_summary.csv b/reports/old/tables/1h/score_summary.csv
similarity index 100%
rename from reports/tables/1h/score_summary.csv
rename to reports/old/tables/1h/score_summary.csv
diff --git a/reports/tables/1h/tasks_leaderboard.csv b/reports/old/tables/1h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/1h/tasks_leaderboard.csv
rename to reports/old/tables/1h/tasks_leaderboard.csv
diff --git a/reports/tables/4h/all_results.csv b/reports/old/tables/4h/all_results.csv
similarity index 100%
rename from reports/tables/4h/all_results.csv
rename to reports/old/tables/4h/all_results.csv
diff --git a/reports/tables/4h/folds_leaderboard.csv b/reports/old/tables/4h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/4h/folds_leaderboard.csv
rename to reports/old/tables/4h/folds_leaderboard.csv
diff --git a/reports/tables/4h/models_summary.csv b/reports/old/tables/4h/models_summary.csv
similarity index 100%
rename from reports/tables/4h/models_summary.csv
rename to reports/old/tables/4h/models_summary.csv
diff --git a/reports/tables/4h/norm_score_summary.csv b/reports/old/tables/4h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/4h/norm_score_summary.csv
rename to reports/old/tables/4h/norm_score_summary.csv
diff --git a/reports/tables/4h/result_summary.csv b/reports/old/tables/4h/result_summary.csv
similarity index 100%
rename from reports/tables/4h/result_summary.csv
rename to reports/old/tables/4h/result_summary.csv
diff --git a/reports/tables/4h/score_summary.csv b/reports/old/tables/4h/score_summary.csv
similarity index 100%
rename from reports/tables/4h/score_summary.csv
rename to reports/old/tables/4h/score_summary.csv
diff --git a/reports/tables/4h/tasks_leaderboard.csv b/reports/old/tables/4h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/4h/tasks_leaderboard.csv
rename to reports/old/tables/4h/tasks_leaderboard.csv
diff --git a/reports/tables/8h/all_results.csv b/reports/old/tables/8h/all_results.csv
similarity index 100%
rename from reports/tables/8h/all_results.csv
rename to reports/old/tables/8h/all_results.csv
diff --git a/reports/tables/8h/folds_leaderboard.csv b/reports/old/tables/8h/folds_leaderboard.csv
similarity index 100%
rename from reports/tables/8h/folds_leaderboard.csv
rename to reports/old/tables/8h/folds_leaderboard.csv
diff --git a/reports/tables/8h/models_summary.csv b/reports/old/tables/8h/models_summary.csv
similarity index 100%
rename from reports/tables/8h/models_summary.csv
rename to reports/old/tables/8h/models_summary.csv
diff --git a/reports/tables/8h/norm_score_summary.csv b/reports/old/tables/8h/norm_score_summary.csv
similarity index 100%
rename from reports/tables/8h/norm_score_summary.csv
rename to reports/old/tables/8h/norm_score_summary.csv
diff --git a/reports/tables/8h/result_summary.csv b/reports/old/tables/8h/result_summary.csv
similarity index 100%
rename from reports/tables/8h/result_summary.csv
rename to reports/old/tables/8h/result_summary.csv
diff --git a/reports/tables/8h/score_summary.csv b/reports/old/tables/8h/score_summary.csv
similarity index 100%
rename from reports/tables/8h/score_summary.csv
rename to reports/old/tables/8h/score_summary.csv
diff --git a/reports/tables/8h/tasks_leaderboard.csv b/reports/old/tables/8h/tasks_leaderboard.csv
similarity index 100%
rename from reports/tables/8h/tasks_leaderboard.csv
rename to reports/old/tables/8h/tasks_leaderboard.csv
diff --git a/reports/readme.md b/reports/readme.md
new file mode 100644
index 000000000..4dee98d67
--- /dev/null
+++ b/reports/readme.md
@@ -0,0 +1,4 @@
+# Notebooks for Visualizing AMLB Results
+
+This folder will contain the notebooks used to visualize result files produced by the AMLB tool, including those notebooks used for generating figures and tables in our paper.
+These notebooks will be added soon.
diff --git a/requirements.in b/requirements.in
index f4f3f772b..19adef7d2 100644
--- a/requirements.in
+++ b/requirements.in
@@ -1,10 +1,13 @@
 boto3>=1.9,<2.0
 liac-arff>=2.5,<3.0
-numpy>=1.20,<2.0
-pandas>=1.2.4,<2.0
 psutil>=5.4,<6.0
 ruamel.yaml>=0.15,<1.0
-openml==0.12.2
-scikit-learn>=0.24
-pyarrow>=4.0
-tables>=3.6
+filelock>=3.4
+
+numpy>=1.24,<2.0
+pandas>=1.5,<2.0
+openml==0.13.1
+scikit-learn>=1.0,<2.0
+
+pyarrow>=11.0
+# tables>=3.6
diff --git a/requirements.txt b/requirements.txt
index 2c2dd4a90..52d6e87a6 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,92 +1,88 @@
 #
-# This file is autogenerated by pip-compile
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
 #
-#    pip-compile requirements.in
+#    pip-compile --output-file=requirements.txt requirements.in
 #
-boto3==1.17.74
+boto3==1.26.98
     # via -r requirements.in
-botocore==1.20.74
+botocore==1.29.98
     # via
     #   boto3
     #   s3transfer
-certifi==2020.12.5
+certifi==2022.12.7
     # via
     #   minio
     #   requests
-chardet==4.0.0
+charset-normalizer==3.1.0
     # via requests
-idna==2.10
+filelock==3.12.0
+    # via -r requirements.in
+idna==3.4
     # via requests
-jmespath==0.10.0
+jmespath==1.0.1
     # via
     #   boto3
     #   botocore
-joblib==1.0.1
+joblib==1.2.0
     # via scikit-learn
 liac-arff==2.5.0
     # via
     #   -r requirements.in
     #   openml
-minio==7.0.3
+minio==7.1.13
     # via openml
-numexpr==2.7.3
-    # via tables
-numpy==1.20.3
+numpy==1.24.2
     # via
     #   -r requirements.in
-    #   numexpr
     #   openml
     #   pandas
     #   pyarrow
     #   scikit-learn
     #   scipy
-    #   tables
-openml==0.12.2
+openml==0.13.1
     # via -r requirements.in
-pandas==1.2.4
+pandas==1.5.3
     # via
     #   -r requirements.in
     #   openml
-psutil==5.8.0
+psutil==5.9.4
     # via -r requirements.in
-pyarrow==4.0.0
+pyarrow==11.0.0
     # via
     #   -r requirements.in
     #   openml
-python-dateutil==2.8.1
+python-dateutil==2.8.2
     # via
     #   botocore
     #   openml
     #   pandas
-pytz==2021.1
+pytz==2022.7.1
     # via pandas
-requests==2.25.1
+requests==2.28.2
     # via openml
-ruamel.yaml.clib==0.2.2
-    # via ruamel.yaml
-ruamel.yaml==0.17.4
+ruamel-yaml==0.17.21
     # via -r requirements.in
-s3transfer==0.4.2
+ruamel-yaml-clib==0.2.7
+    # via ruamel-yaml
+s3transfer==0.6.0
     # via boto3
-scikit-learn==0.24.2
+scikit-learn==1.2.2
     # via
     #   -r requirements.in
     #   openml
-scipy==1.6.3
+scipy==1.10.1
     # via
     #   openml
     #   scikit-learn
 six==1.16.0
     # via python-dateutil
-tables==3.6.1
-    # via -r requirements.in
-threadpoolctl==2.1.0
+threadpoolctl==3.1.0
     # via scikit-learn
-urllib3==1.26.4
+urllib3==1.26.15
     # via
     #   botocore
     #   minio
     #   requests
-xmltodict==0.12.0
+xmltodict==0.13.0
     # via openml
diff --git a/resources/benchmarks/timeseries.yaml b/resources/benchmarks/timeseries.yaml
new file mode 100644
index 000000000..26af06497
--- /dev/null
+++ b/resources/benchmarks/timeseries.yaml
@@ -0,0 +1,13 @@
+---
+
+- name: covid
+  dataset:
+    train: https://autogluon.s3-us-west-2.amazonaws.com/datasets/CovidTimeSeries/train.csv
+    test: https://autogluon.s3-us-west-2.amazonaws.com/datasets/CovidTimeSeries/test.csv
+    target: ConfirmedCases
+    type: timeseries
+    forecast_range_in_steps: 19
+    id_column: name
+    timestamp_column: Date
+
+  folds: 1
diff --git a/resources/config.yaml b/resources/config.yaml
index c03a742eb..bb7b43f9b 100644
--- a/resources/config.yaml
+++ b/resources/config.yaml
@@ -33,7 +33,7 @@ frameworks:              # configuration namespace for the frameworks definition
     - '{root}/resources/frameworks.yaml'
   root_module: frameworks     # the default python module under which the frameworks modules are defined.
   allow_duplicates: false     # if true, the last definition is used.
-  tags: ['stable', 'latest', '2020Q2', '2021Q3']  # the list of supported tags when looking up frameworks:
+  tags: ['stable', 'latest', '2020Q2', '2021Q3', '2023Q2']  # the list of supported tags when looking up frameworks:
                               # for example frmwk:latest will look for framework frmwk in a frameworks_latest.yaml file if present.
 
 benchmarks:                     # configuration namespace for the benchmarks definitions.
@@ -54,6 +54,7 @@ benchmarks:                     # configuration namespace for the benchmarks def
     binary: ['auc', 'logloss', 'acc', 'balacc']     # available metrics: auc (AUC), acc (Accuracy), balacc (Balanced Accuracy), pr_auc (Precision Recall AUC), logloss (Log Loss), f1, f2, f05 (F-beta scores with beta=1, 2, or 0.5), max_pce, mean_pce (Max/Mean Per-Class Error).
     multiclass: ['logloss', 'acc', 'balacc']        # available metrics: same as for binary, except auc, replaced by auc_ovo (AUC One-vs-One), auc_ovr (AUC One-vs-Rest). AUC metrics and F-beta metrics are computed with weighted average.
     regression: ['rmse', 'r2', 'mae']               # available metrics: mae (Mean Absolute Error), mse (Mean Squared Error), msle (Mean Squared Logarithmic Error), rmse (Root Mean Square Error), rmsle (Root Mean Square Logarithmic Error), r2 (R^2).
+    timeseries: ['mase', 'mape', 'smape', 'rmse', 'mse', 'nrmse', 'wape', 'ncrps']
   defaults:            # the default constraints, usually overridden by a constraint.
     folds: 10          # the amount of fold-runs executed for each dataset.
     max_runtime_seconds: 3600   # default time allocated to the framework to train a model.
@@ -79,7 +80,17 @@ monitoring:               # configuration namespace describing the basic monitor
 
 results:                 # configuration namespace for the results.csv file.
   error_max_length: 200  # the max length of the error message as rendered in the results file.
-  save: true             # set by runbenchmark.py
+  global_save: true      # set by runbenchmark.py, if true adds the results to the main `results.csv` in the {output.dir}
+  global_lock_timeout: 5 # the timeout used to wait for the lock on the global results file.
+  incremental_save: true # if true save results after each job., otherwise save results only when all jobs are completed.
+
+inference_time_measurements:  # configuration namespace for performing additional inference time measurements on various batch sizes
+  enabled: false
+  batch_sizes: [1, 10, 100, 1000, 10000]  # the batch sizes for which inference speed should be measured
+  repeats: 10                            # the number of times to repeat the inference measurement for each batch size
+  additional_job_time: 300  # the time in seconds that will be added to the maximum job time if inference time is measured
+  limit_by_dataset_size: true  # Don't measure inference time on `batch size` if it exceeds the number of rows in the dataset.
+                               # E.g., on micro-mass (571 rows) with `batch_sizes` [1, 10, 100, 1000, 10000], only measure [1, 10, 100].
 
 openml:                # configuration namespace for openML.
   apikey: c1994bdb7ecb3c6f3c8f3b35f4b47f1f
@@ -87,7 +98,7 @@ openml:                # configuration namespace for openML.
 
 versions:              # configuration namespace for versions enforcement (libraries versions are usually enforced in requirements.txt for the app and for each framework).
   pip:
-  python: 3.7          # the Python minor version that will be used by the application in containers and cloud instances, , also used as a based version for virtual environments created for each framework.
+  python: 3.9          # the Python minor version that will be used by the application in containers and cloud instances, also used as a based version for virtual environments created for each framework.
 
 container: &container          # parent configuration namespace for container modes.
   force_branch: true           # set to true if image can only be built from a clean branch, with same tag as defined in `project_repository`.
@@ -103,6 +114,13 @@ container: &container          # parent configuration namespace for container mo
 docker:                        # configuration namespace for docker: it inherits from `container` namespace.
   <<: *container
   run_extra_options: '--shm-size=2048M'
+  run_as: 'default'    # Sets the user inside the docker container (`docker run -u`), one of:
+                       #  * 'user': set as `-u $(id -u):$(id -g)`, only on unix systems.
+                       #  * 'root': set as `-u 0:0`
+                       #  * 'default': does not set `-u`
+                       #  * any string that starts with `-u`, which will be directly forwarded.
+                       # Try this setting if you have problems with permissions in docker.
+  build_extra_options: ''
 
 singularity:                   # configuration namespace for docker: it inherits from `container` namespace.
   <<: *container
@@ -137,17 +155,20 @@ aws:                    # configuration namespace for AWS mode.
   ec2:
     regions:                             #
       us-east-1:
-        ami: ami-0ac019f4fcb7cb7e6
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-053b0d53c279acc90
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
+      us-east-2:
+        ami: ami-024e6efaf93d85776
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       us-west-1:
-        ami: ami-063aa838bd7631e0b
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-0f8e81a3da6e2510a
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       eu-west-1:
-        ami: ami-00035f41c82244dab
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-01dd271720c1ba44f
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
       eu-central-1:
-        ami: ami-0bdf93799014acdc4
-        description: Ubuntu Server 18.04 LTS (HVM), EBS General Purpose (SSD) VolumeType
+        ami: ami-04e601abe3e1a910f
+        description: Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-05-16
     instance_type:                       #
       series: m5                         #
       map:                               # map between num cores required and ec2 instance type sizes
@@ -157,7 +178,9 @@ aws:                    # configuration namespace for AWS mode.
         '4': xlarge
         '8': 2xlarge
         '16': 4xlarge
+    instance_tags: {}                    # specify custom tags for the EC2 instances here
     volume_type: standard                # one of gp2, io1, st1, sc1, or standard (default).
+    volume_tags: {}                      # specify custom tags for the volume tags here (if empty, will apply the same tags as the corresponding instance)
     root_device_name: '/dev/sda1'        #
     availability_zone:                   # the availability zone where the instances will be created (if not set, aws will pick a default one).
     subnet_id: ''                        #
diff --git a/resources/frameworks.yaml b/resources/frameworks.yaml
index eb3b7d9e4..74eeee805 100644
--- a/resources/frameworks.yaml
+++ b/resources/frameworks.yaml
@@ -32,9 +32,26 @@ AutoGluon_bestquality:
   extends: AutoGluon
   description: |
     AutoGluon with 'best_quality' preset provides the most accurate overall predictor.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  description: |
+    AutoGluon with 'high_quality' preset provides a very strong predictor with 8x+ faster inference speed than 'best_quality'.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  description: |
+    AutoGluon with 'good_quality' preset provides a strong predictor with 16x+ faster inference speed than 'best_quality'.
+    Refer to https://auto.gluon.ai/stable/tutorials/tabular_prediction/tabular-quickstart.html#presets for a description of the presets.
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'stable'
   description: |
@@ -60,7 +77,8 @@ AutoWEKA:
   refs: [https://www.cs.ubc.ca/labs/beta/Projects/autoweka/papers/16-599.pdf]
 
 autoxgboost:
-  version: 'latest'
+  removed: |
+    Autoxgboost is no longer supported. The last confirmed working version was at v2.0.2 with docker image 'automlbenchmark/autoxgboost:908631665f4c763d548ed203d48f05fe68613844-v2.0.2'.
   description: |
     autoxgboost aims to find an optimal xgboost model automatically using the machine learning framework mlr and the bayesian optimization framework mlrMBO.
   project: https://github.com/ja-thomas/autoxgboost
@@ -69,9 +87,9 @@ autoxgboost:
 flaml:
   version: 'stable'
   description: |
-    FLAML is a lightweight Python library that finds accurate machine learning models 
-    automatically, efficiently and economically. It frees users from selecting learners 
-    and hyperparameters for each learner. It is fast and cheap. 
+    FLAML is a lightweight Python library that finds accurate machine learning models
+    automatically, efficiently and economically. It frees users from selecting learners
+    and hyperparameters for each learner. It is fast and cheap.
   project: https://github.com/microsoft/FLAML
   refs: [https://arxiv.org/pdf/1911.04706.pdf]
 
@@ -122,12 +140,12 @@ mljarsupervised_compete:
   description: "MLJAR is using 'Compete' mode to provide the most accurate predictor"
   params:
     mode: Compete   # set mode for Compete, default mode is Explain
-    
+
 MLNet:
   version: 'latest'
   description: |
     MLNET.CLI is a automated machine learning tool implemented by ml.net.
-    
+
 MLPlan:
   version: 'stable'
   abstract: true
@@ -174,7 +192,17 @@ TPOT:
 #    population_size: 25
 #    verbosity: 2
 
+####################################
+### TimeSeries AutoML frameworks ###
+####################################
 
+AutoGluonTS:
+  extends: AutoGluon
+  version: "stable"
+  description: |
+    AutoGluon-TimeSeries
+  setup_env:
+    MODULE: timeseries
 
 #######################################
 ### Non AutoML reference frameworks ###
@@ -217,4 +245,3 @@ TunedRandomForest:
 #    _n_jobs: 1  # cf. RandomForest
 #    _tuning:
 #      n_estimators: 500
-
diff --git a/resources/frameworks_2020Q2.yaml b/resources/frameworks_2020Q2.yaml
index 218827276..b8ace6c21 100644
--- a/resources/frameworks_2020Q2.yaml
+++ b/resources/frameworks_2020Q2.yaml
@@ -19,9 +19,6 @@ autosklearn:
 AutoWEKA:
   version: '2.6'
 
-autoxgboost:
-  version: 'latest'
-
 GAMA:
   version: '20.1.0'
 
diff --git a/resources/frameworks_2021Q3.yaml b/resources/frameworks_2021Q3.yaml
index cea36f5ba..ad1b42e8e 100644
--- a/resources/frameworks_2021Q3.yaml
+++ b/resources/frameworks_2021Q3.yaml
@@ -22,9 +22,6 @@ autosklearn2:
   params:
     _askl2: true
 
-autoxgboost:
-  version: '#908631665f4c763d548ed203d48f05fe68613844'
-
 AutoWEKA:
   version: '2.6'
 
diff --git a/resources/frameworks_2023Q2.yaml b/resources/frameworks_2023Q2.yaml
new file mode 100644
index 000000000..8923bf246
--- /dev/null
+++ b/resources/frameworks_2023Q2.yaml
@@ -0,0 +1,119 @@
+---
+
+#########################
+### AutoML frameworks ###
+#########################
+
+AutoGluon:
+  abstract: true
+  version: "0.8.0"
+
+AutoGluon_benchmark:
+  extends: AutoGluon
+  description: "AutoGluon with 'best_quality' preset provides the most accurate overall predictor"
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: best_quality
+    _leaderboard_test: True
+
+AutoGluon_hq:
+  extends: AutoGluon
+  description: "AutoGluon with 'high_quality' preset provides generally fast inference speed with high accuracy"
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: ['high_quality', 'optimize_for_deployment']
+    _leaderboard_test: True
+
+AutoGluon_hq_il001:
+  extends: AutoGluon
+  description: "AutoGluon ~3x faster inference at slight performance loss to 'high quality' (self-reported)."
+  params:
+    _save_artifacts: ['leaderboard', 'info']
+    presets: ['high_quality', 'optimize_for_deployment']
+    _leaderboard_test: True
+    infer_limit: 0.01
+
+autosklearn:
+  version: '0.15.0'
+
+autosklearn2:
+  extends: autosklearn
+  params:
+    _askl2: true
+
+AutoWEKA:
+  version: '2.6'
+
+flaml:
+  version: '1.2.4'
+
+GAMA:
+  abstract: true
+  version: '23.0.0'
+
+GAMA_benchmark:
+  extends: GAMA
+  params:
+    preset: performance
+
+H2OAutoML:
+  version: '3.40.0.4'
+
+lightautoml:
+  version: '0.3.7.3'
+
+mljarsupervised:
+  abstract: true
+  version: '0.11.5'
+
+mljarsupervised_benchmark:
+  extends: mljarsupervised
+  description: "MLJar using 'Compete' mode to provide most accurate predictor"
+  params:
+    mode: Compete
+
+mljarsupervised_perform:
+  extends: mljarsupervised
+  description: "MLJar using 'Perform' mode to provide fast inference"
+  params:
+    mode: Perform
+
+MLPlan:
+  abstract: true
+  version: '0.2.5'
+
+MLPlanWEKA:
+  extends: MLPlan
+  params:
+    _backend: weka
+
+mlr3automl:
+  version: '#f667900292e3ded64bb419285c71cd5d1d2c4301'
+  project: https://github.com/a-hanf/mlr3automl
+
+TPOT:
+  version: '0.12.0'
+
+
+#######################################
+### Non AutoML reference frameworks ###
+#######################################
+
+constantpredictor: {}
+
+DecisionTree:
+  version: '1.2.2'
+
+RandomForest:
+  version: '1.2.2'
+  params:
+    n_estimators: 2000
+
+ranger:
+  version: 'stable'
+
+TunedRandomForest:
+  version: '1.2.2'
+  params:
+    n_estimators: 2000
+
diff --git a/resources/frameworks_stable.yaml b/resources/frameworks_stable.yaml
index ff55160bf..15c6deb33 100644
--- a/resources/frameworks_stable.yaml
+++ b/resources/frameworks_stable.yaml
@@ -15,6 +15,16 @@ AutoGluon_bestquality:
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'stable'
 
@@ -26,15 +36,24 @@ autosklearn2:
 AutoWEKA:
   version: 'stable'
 
-autoxgboost:
+
+flaml:
   version: 'stable'
 
+
+flaml_benchmark:
+  extends: flaml
+  version: "benchmark"
+
 GAMA:
   version: 'stable'
 
 H2OAutoML:
   version: 'stable'
 
+lightautoml:
+  version: 'stable'
+
 hyperoptsklearn:
   version: 'stable'
 
@@ -60,6 +79,9 @@ MLPlanWEKA:
   params:
     _backend: weka
 
+mlr3automl:
+  version: 'stable'
+
 oboe:
   version: 'stable'
 
diff --git a/runbenchmark.py b/runbenchmark.py
index 7ff8c6065..ad5eac7e3 100644
--- a/runbenchmark.py
+++ b/runbenchmark.py
@@ -8,19 +8,12 @@
 # prevent asap other modules from defining the root logger using basicConfig
 import amlb.logger
 
-from openml.config import cache_directory
+import openml
 
 import amlb
 from amlb.utils import Namespace as ns, config_load, datetime_iso, str2bool, str_sanitize, zip_path
 from amlb import log, AutoMLError
-
-
-default_dirs = ns(
-    input_dir=cache_directory,
-    output_dir="./results",
-    user_dir="~/.config/automlbenchmark",
-    root_dir=os.path.dirname(__file__)
-)
+from amlb.defaults import default_dirs
 
 parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
 parser.add_argument('framework', type=str,
@@ -31,7 +24,8 @@
                     help="The benchmark type to run as defined by default in resources/benchmarks/{benchmark}.yaml,"
                          "\na path to a benchmark description file, or an openml suite or task."
                          "\nOpenML references should be formatted as 'openml/s/X' and 'openml/t/Y',"
-                         "\nfor studies and tasks respectively."
+                         "\nfor studies and tasks respectively. Use 'test.openml/s/X' for the "
+                         "\nOpenML test server."
                          "\n(default: '%(default)s')")
 parser.add_argument('constraint', type=str, nargs='?', default='test',
                     help="The constraint definition to use as defined by default in resources/constraints.yaml."
@@ -84,6 +78,11 @@
                          "\n  --logging=root:debug (keeps defaults for non-specified loggers)" 
                          "\n  --logging=console:warning,app:info"
                          "\n(default: '%(default)s')")
+parser.add_argument('--openml-test-server', type=str2bool, metavar='true|false', nargs='?', const=True, default=False,
+                    help=argparse.SUPPRESS)  # "Set to true to connect to the OpenML test server instead."
+parser.add_argument('--openml-run-tag', type=str, default=None,
+                    help="Tag that will be saved in metadata and OpenML runs created during upload, must match '([a-zA-Z0-9_\-\.])+'.")
+
 parser.add_argument('--profiling', nargs='?', const=True, default=False, help=argparse.SUPPRESS)
 parser.add_argument('--resume', nargs='?', const=True, default=False, help=argparse.SUPPRESS)
 parser.add_argument('--session', type=str, default=None, help=argparse.SUPPRESS)
@@ -119,7 +118,7 @@
 # now_str = datetime_iso(time=False, no_sep=True)
 if args.profiling:
     logging.TRACE = logging.INFO
-log_levels = ns({logger: level.upper()
+log_levels = ns({logger: int(level) if level.isnumeric() else level.upper()
                  for logger, level in [d.split(':') for d in args.logging.split(',')]} if ':' in args.logging
                 else dict(console=args.logging.upper(), app=args.logging.upper(), root=args.logging.upper()) if args.logging
                 else {}) | ns(console='INFO', app='DEBUG', root='INFO')  # adding defaults if needed
@@ -128,6 +127,10 @@
                   root_level=log_levels.root, app_level=log_levels.app, console_level=log_levels.console, print_to_log=True)
 
 log.info("Running benchmark `%s` on `%s` framework in `%s` mode.", args.framework, args.benchmark, args.mode)
+if args.openml_test_server:
+    openml.config.start_using_configuration_for_example()
+    log.info("Connecting to the OpenML test server.")
+
 log.debug("Script args: %s.", args)
 
 config_default = config_load(os.path.join(default_dirs.root_dir, "resources", "config.yaml"))
@@ -136,7 +139,7 @@
 config_user = config_load(extras.get('config', os.path.join(args.userdir or default_dirs.user_dir, "config.yaml")))
 # config listing properties set by command line
 config_args = ns.parse(
-    {'results.save': args.keep_scores},
+    {'results.global_save': args.keep_scores},
     input_dir=args.indir,
     output_dir=args.outdir,
     user_dir=args.userdir,
@@ -145,6 +148,9 @@
     parallel_jobs=args.parallel,
     sid=sid,
     exit_on_error=args.exit_on_error,
+    test_server=args.openml_test_server,
+    tag=args.openml_run_tag,
+    command=' '.join(sys.argv),
 ) + ns.parse(extras)
 if args.mode != 'local':
     config_args + ns.parse({'monitoring.frequency_seconds': 0})
@@ -155,8 +161,8 @@
 if args.resume:
     amlb_res.config.job_history = os.path.join(amlb_res.config.output_dir, amlb.results.Scoreboard.results_file)
 
-code = 0
 bench = None
+exit_code = 0
 try:
     if args.mode == 'local':
         bench = amlb.Benchmark(args.framework, args.benchmark, args.constraint)
@@ -185,10 +191,10 @@
     log.error('\nERROR:\n%s', e)
     if extras.get('verbose') is True:
         log.exception(e)
-    code = 1
+    exit_code = 1
 except Exception as e:
     log.exception(e)
-    code = 2
+    exit_code = 2
 finally:
     archives = amlb.resources.config().archive
     if archives and bench:
@@ -197,5 +203,7 @@
             if d in out_dirs:
                 zip_path(out_dirs[d], os.path.join(out_dirs.session, f"{d}.zip"), arc_path_format='long')
                 shutil.rmtree(out_dirs[d], ignore_errors=True)
+    if args.openml_test_server:
+        openml.config.stop_using_configuration_for_example()
 
-    sys.exit(code)
+    sys.exit(exit_code)
diff --git a/runstable.sh b/runstable.sh
index 28d8628f9..3be2857e6 100755
--- a/runstable.sh
+++ b/runstable.sh
@@ -12,7 +12,6 @@ h2oautoml
 tpot
 oboe
 autoweka
-autoxgboost
 #hyperoptsklearn
 #ranger
 )
diff --git a/upload_results.py b/upload_results.py
new file mode 100644
index 000000000..800cde0f7
--- /dev/null
+++ b/upload_results.py
@@ -0,0 +1,188 @@
+""" Script to upload results from the benchmark to OpenML.
+The benchmark run needs to be executed on OpenML datasets to be eligible for upload.
+"""
+import argparse
+import contextlib
+from contextlib import contextmanager
+from datetime import datetime
+import logging
+import os
+import pathlib
+from typing import Optional
+
+import openml
+from openml import OpenMLRun
+
+from amlb.defaults import default_dirs
+from amlb.resources import config_load
+from amlb.uploads import process_task_folder, missing_folds, _load_task_data
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.DEBUG)
+
+
+def parse_args():
+    description = "Script to upload results from the benchmark to OpenML."
+    parser = argparse.ArgumentParser(description)
+    parser.add_argument(
+        '-i', '--input-directory', type=pathlib.Path, default=None, dest='input_directory',
+        help="Directory that stores results from the runbenchmark.py invocation. "
+             "By default use the most recent folder in the results "
+             "folder as specified in the configuration."
+    )
+    parser.add_argument(
+        '-a', '--api-key', dest='apikey', default=None,
+        help="By default, the api key configured in your OpenML configuration file is"
+             "used. Specify this parameter if you want to overwrite this value or"
+             "you do not have an OpenML configuration file. A valid key is "
+             "*required* to upload to the OpenML server."
+             "This argument is ignored when uploading to the test server, as "
+             "the default openml-python test server api key will be used instead."
+    )
+    parser.add_argument(
+        '-m', '--mode', dest='mode', default='check',
+        help="Run mode (default=%(default)s)."
+             "• check: only report whether results can be uploaded."
+             "• upload: upload all complete results."
+    )
+    parser.add_argument(
+        '-x', '--fail-fast', dest='fail_fast', action='store_true',
+        help="Stop as soon as a task fails to upload due to an error during uploading."
+    )
+    parser.add_argument(
+        '-v', '--verbose', action='store_true', dest='verbose',
+        help="Output progress to console."
+    )
+    parser.add_argument(
+        '-t', '--task', type=str, dest='task', default=None,
+        help="Only upload results for this specific task."
+    )
+    args = parser.parse_args()
+
+    if args.mode not in ['check', 'upload']:
+        raise ValueError(f"Invalid value for argument 'mode': '{args.mode}'.")
+
+    return args
+
+
+def find_most_recent_result_folder() -> pathlib.Path:
+    root_dir = pathlib.Path(__file__).parent
+    config = config_load(root_dir / "resources" / "config.yaml")
+    output_dir = pathlib.Path(config.output_dir or default_dirs.output_dir)
+
+    def dirname_to_datetime(dirname: str) -> datetime:
+        _, timestamp = dirname.rsplit('.', 1)
+        return datetime.strptime(timestamp, "%Y%m%dT%H%M%S")
+
+    run_directories = output_dir.glob("*.*.*.*")
+    _, run_directory = max((dirname_to_datetime(str(d)), d) for d in run_directories)
+    return run_directory
+
+
+def resolve_input_directory(path: Optional[pathlib.Path]) -> pathlib.Path:
+    path = path or find_most_recent_result_folder()
+    path = path.expanduser().absolute()
+    if not path.is_dir():
+        raise ValueError(f"{path} is not a directory.")
+    return path
+
+
+def configure_logging(verbose: bool):
+    formatter = logging.Formatter(
+        '[%(asctime)s] %(levelname)s: %(message)s',
+        '%H:%M:%S',
+    )
+    log_level = logging.DEBUG if verbose else logging.INFO
+
+    console = logging.StreamHandler()
+    console.setFormatter(formatter)
+    console.setLevel(log_level)
+
+    log.addHandler(console)
+
+
+@contextmanager
+def _connect_to_test_server():
+    openml.config.start_using_configuration_for_example()
+    yield
+    openml.config.stop_using_configuration_for_example()
+
+
+def server_for_task(task: pathlib.Path):
+    metadata = _load_task_data(task)
+    if metadata.test_server:
+        server_connection = _connect_to_test_server()
+    else:
+        server_connection = contextlib.nullcontext()
+    return server_connection
+
+
+def upload_task(task_directory: pathlib.Path) -> Optional[OpenMLRun]:
+    task_name = os.path.basename(task_directory)
+    try:
+        with server_for_task(task_directory):
+            log.debug("Starting upload for '%s'." % task_name)
+            run = process_task_folder(task_directory)
+            log.info("%s result stored at %s/r/%d"
+                     % (task_name, openml.config.server[:-11], run.id))
+            return run
+    except Exception as e:
+        message = e.message if hasattr(e, "message") else e.args[0]
+        log.warning("Task %s failed to upload: %s" % (task_name, message))
+        if args.fail_fast:
+            raise
+
+
+def process_results(result_dir: pathlib.Path, mode: str = 'check'):
+    prediction_directory = result_dir / "predictions"
+    if not prediction_directory.exists():
+        raise ValueError(f"result_dir '{result_dir!s}' has no predictions.")
+
+    if args.task is None:
+        tasks_to_process = [d for d in prediction_directory.iterdir() if d.is_dir()]
+    elif (prediction_directory / args.task).is_dir():
+        tasks_to_process = [args.task]
+    else:
+        raise ValueError(f"Task '{args.task}' not in '{prediction_directory}'.")
+
+    for task_name in tasks_to_process:
+        full_task_directory = prediction_directory / task_name
+
+        folds = missing_folds(full_task_directory)
+        if len(folds) > 0:
+            log.info("%s has missing folds: %s" % (task_name, ', '.join(sorted(folds))))
+            continue
+
+        metadata = _load_task_data(full_task_directory)
+        if "openml_task_id" not in metadata:
+            log.info("%s has openml task metadata" % task_name)
+            continue
+
+        if mode == 'check':
+            log.info("%s is ready for upload." % task_name)
+        elif mode == 'upload':
+            upload_task(full_task_directory)
+
+
+def configure_apikey(key: Optional[str]) -> bool:
+    if key:
+        openml.config.apikey = key
+    # API does not support easy checking of validity
+    is_valid = True
+    return openml.config.apikey is not None and is_valid
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    configure_logging(args.verbose)
+    valid_key = configure_apikey(args.apikey)
+    if not valid_key and args.mode == 'upload':
+        raise ValueError(
+            "No valid OpenML API key configured, use the '--api-key' argument "
+            "or follow instructions: https://openml.github.io/openml-python/master/usage.html#configuration"
+        )
+    input_directory = resolve_input_directory(args.input_directory)
+    mode_verb = 'Uploading' if args.mode == 'upload' else 'Checking'
+    log.info("%s results from '%s'." % (mode_verb, input_directory))
+
+    process_results(input_directory, args.mode)


Test patch:
diff --git a/.github/workflows/run_pytest.yaml b/.github/workflows/run_pytest.yaml
new file mode 100644
index 000000000..52be3c836
--- /dev/null
+++ b/.github/workflows/run_pytest.yaml
@@ -0,0 +1,35 @@
+name: Light-weight Unit Tests
+
+on: pull_request
+
+
+jobs:
+  pytest_not_stress:
+    name: pytest
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+      - name: Create venv
+        run: python -m venv venv
+      - uses: actions/cache@v3
+        id: cache
+        with:
+          path: /home/runner/work/automlbenchmark/automlbenchmark/venv/lib/python3.9/site-packages
+          key: pip-v3-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/requirements-dev.txt') }}
+          restore-keys: |
+            pip-v3-
+      - name: Install Requirements
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: |
+          source venv/bin/activate
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+      - name: Run unit tests
+        run: |
+          source venv/bin/activate
+          python -m pytest -vv -m "not stress and not openml_upload"
diff --git a/resources/frameworks_latest.yaml b/resources/frameworks_latest.yaml
index 0a4705d96..adb90f360 100644
--- a/resources/frameworks_latest.yaml
+++ b/resources/frameworks_latest.yaml
@@ -16,6 +16,16 @@ AutoGluon_bestquality:
   params:
     presets: best_quality
 
+AutoGluon_hq:
+  extends: AutoGluon
+  params:
+    presets: high_quality
+
+AutoGluon_gq:
+  extends: AutoGluon
+  params:
+    presets: good_quality
+
 autosklearn:
   version: 'latest'
 
@@ -27,9 +37,6 @@ autosklearn2:
 AutoWEKA:
   version: 'latest'
 
-autoxgboost:
-  version: 'latest'
-
 GAMA:
   version: 'latest'
 
@@ -50,6 +57,9 @@ mljarsupervised_compete:
   params:
     mode: Compete   # set mode for Compete, default mode is Explain
 
+MLNet:
+  version: 'latest'
+
 MLPlan:
   abstract: true
   version: 'latest'
@@ -64,14 +74,26 @@ MLPlanWEKA:
   params:
     _backend: weka
 
+mlr3automl:
+  version: 'latest'
+
 oboe:
   version: 'latest'
 
 TPOT:
   version: 'latest'
 
+####################################
+### TimeSeries AutoML frameworks ###
+####################################
 
-
+AutoGluonTS:
+  extends: AutoGluon
+  version: "latest"
+  description: |
+    AutoGluon-TimeSeries
+  setup_env:
+    MODULE: timeseries
 
 #######################################
 ### Non AutoML reference frameworks ###
@@ -94,4 +116,3 @@ TunedRandomForest:
   version: 'latest'
   params:
     n_estimators: 2000
-
diff --git a/tests/unit/amlb/datasets/file/resources/iris_num_test.csv b/tests/unit/amlb/datasets/file/resources/iris_num_test.csv
new file mode 100644
index 000000000..af6e38257
--- /dev/null
+++ b/tests/unit/amlb/datasets/file/resources/iris_num_test.csv
@@ -0,0 +1,16 @@
+sepallength,sepalwidth,petallength,petalwidth,class
+5.0,3.5,1.6,0.6,1
+5.8,4.0,1.2,0.2,1
+4.9,3.1,1.5,0.1,1
+5.1,3.3,1.7,0.5,1
+5.4,3.7,1.5,0.2,1
+5.7,2.8,4.1,1.3,2
+6.3,2.3,4.4,1.3,2
+6.2,2.9,4.3,1.3,2
+6.0,2.2,4.0,1.0,2
+5.8,2.6,4.0,1.2,2
+6.0,2.2,5.0,1.5,3
+6.4,2.7,5.3,1.9,3
+6.7,3.3,5.7,2.5,3
+6.5,3.0,5.5,1.8,3
+7.2,3.2,6.0,1.8,3
diff --git a/tests/unit/amlb/datasets/file/resources/iris_num_train.csv b/tests/unit/amlb/datasets/file/resources/iris_num_train.csv
new file mode 100644
index 000000000..17c83f6f1
--- /dev/null
+++ b/tests/unit/amlb/datasets/file/resources/iris_num_train.csv
@@ -0,0 +1,136 @@
+sepallength,sepalwidth,petallength,petalwidth,class
+5.0,3.3,1.4,0.2,1
+5.7,4.4,1.5,0.4,1
+4.6,3.2,1.4,0.2,1
+5.1,3.5,1.4,0.2,1
+5.1,3.8,1.9,0.4,1
+6.8,2.8,4.8,1.4,2
+5.5,2.5,4.0,1.3,2
+6.6,2.9,4.6,1.3,2
+6.5,2.8,4.6,1.5,2
+5.5,2.3,4.0,1.3,2
+5.9,3.0,5.1,1.8,3
+5.8,2.8,5.1,2.4,3
+6.8,3.0,5.5,2.1,3
+6.4,3.2,5.3,2.3,3
+6.9,3.1,5.4,2.1,3
+4.4,3.2,1.3,0.2,1
+4.5,2.3,1.3,0.3,1
+5.4,3.4,1.5,0.4,1
+5.3,3.7,1.5,0.2,1
+5.0,3.4,1.5,0.2,1
+5.6,2.7,4.2,1.3,2
+5.0,2.3,3.3,1.0,2
+5.7,3.0,4.2,1.2,2
+6.4,3.2,4.5,1.5,2
+6.3,2.5,4.9,1.5,2
+7.7,3.0,6.1,2.3,3
+6.7,3.3,5.7,2.1,3
+6.9,3.1,5.1,2.3,3
+7.6,3.0,6.6,2.1,3
+6.8,3.2,5.9,2.3,3
+4.9,3.1,1.5,0.1,1
+4.6,3.6,1.0,0.2,1
+4.8,3.0,1.4,0.3,1
+4.6,3.1,1.5,0.2,1
+5.4,3.9,1.3,0.4,1
+5.7,2.6,3.5,1.0,2
+5.7,2.8,4.5,1.3,2
+5.1,2.5,3.0,1.1,2
+5.5,2.4,3.8,1.1,2
+6.7,3.0,5.0,1.7,2
+6.9,3.2,5.7,2.3,3
+7.7,2.8,6.7,2.0,3
+6.7,3.0,5.2,2.3,3
+6.3,3.3,6.0,2.5,3
+7.3,2.9,6.3,1.8,3
+5.0,3.6,1.4,0.2,1
+5.2,3.5,1.5,0.2,1
+4.4,3.0,1.3,0.2,1
+5.1,3.8,1.5,0.3,1
+5.5,4.2,1.4,0.2,1
+5.7,2.9,4.2,1.3,2
+7.0,3.2,4.7,1.4,2
+5.9,3.0,4.2,1.5,2
+6.7,3.1,4.4,1.4,2
+6.1,2.8,4.7,1.2,2
+7.2,3.6,6.1,2.5,3
+6.2,2.8,4.8,1.8,3
+6.5,3.2,5.1,2.0,3
+6.2,3.4,5.4,2.3,3
+6.3,2.9,5.6,1.8,3
+5.1,3.8,1.6,0.2,1
+4.6,3.4,1.4,0.3,1
+5.7,3.8,1.7,0.3,1
+5.5,3.5,1.3,0.2,1
+4.8,3.4,1.9,0.2,1
+6.0,2.7,5.1,1.6,2
+5.5,2.6,4.4,1.2,2
+6.2,2.2,4.5,1.5,2
+6.9,3.1,4.9,1.5,2
+6.6,3.0,4.4,1.4,2
+6.1,2.6,5.6,1.4,3
+6.5,3.0,5.8,2.2,3
+7.7,3.8,6.7,2.2,3
+6.1,3.0,4.9,1.8,3
+5.6,2.8,4.9,2.0,3
+5.0,3.4,1.6,0.4,1
+4.3,3.0,1.1,0.1,1
+5.2,4.1,1.5,0.1,1
+5.4,3.4,1.7,0.2,1
+5.1,3.5,1.4,0.3,1
+6.1,2.9,4.7,1.4,2
+5.6,3.0,4.1,1.3,2
+5.6,2.9,3.6,1.3,2
+6.1,3.0,4.6,1.4,2
+5.5,2.4,3.7,1.0,2
+6.4,3.1,5.5,1.8,3
+5.8,2.7,5.1,1.9,3
+7.4,2.8,6.1,1.9,3
+7.7,2.6,6.9,2.3,3
+6.7,2.5,5.8,1.8,3
+4.7,3.2,1.6,0.2,1
+5.0,3.5,1.3,0.3,1
+4.8,3.0,1.4,0.1,1
+5.0,3.2,1.2,0.2,1
+4.4,2.9,1.4,0.2,1
+6.7,3.1,4.7,1.5,2
+5.8,2.7,3.9,1.2,2
+5.4,3.0,4.5,1.5,2
+6.4,2.9,4.3,1.3,2
+4.9,2.4,3.3,1.0,2
+6.7,3.1,5.6,2.4,3
+6.3,2.8,5.1,1.5,3
+4.9,2.5,4.5,1.7,3
+6.4,2.8,5.6,2.1,3
+7.1,3.0,5.9,2.1,3
+5.0,3.0,1.6,0.2,1
+4.9,3.0,1.4,0.2,1
+5.1,3.7,1.5,0.4,1
+4.8,3.1,1.6,0.2,1
+5.2,3.4,1.4,0.2,1
+5.8,2.7,4.1,1.0,2
+6.0,2.9,4.5,1.5,2
+5.6,3.0,4.5,1.5,2
+5.9,3.2,4.8,1.8,2
+5.2,2.7,3.9,1.4,2
+6.3,3.4,5.6,2.4,3
+6.0,3.0,4.8,1.8,3
+7.2,3.0,5.8,1.6,3
+5.7,2.5,5.0,2.0,3
+6.4,2.8,5.6,2.2,3
+5.4,3.9,1.7,0.4,1
+4.8,3.4,1.6,0.2,1
+4.9,3.1,1.5,0.1,1
+5.1,3.4,1.5,0.2,1
+4.7,3.2,1.3,0.2,1
+6.0,3.4,4.5,1.6,2
+6.3,3.3,4.7,1.6,2
+5.6,2.5,3.9,1.1,2
+6.1,2.8,4.0,1.3,2
+5.0,2.0,3.5,1.0,2
+5.8,2.7,5.1,1.9,3
+6.3,2.7,4.9,1.8,3
+7.9,3.8,6.4,2.0,3
+6.5,3.0,5.2,2.0,3
+6.3,2.5,5.0,1.9,3
diff --git a/tests/unit/amlb/datasets/file/test_file_dataloader.py b/tests/unit/amlb/datasets/file/test_file_dataloader.py
index b979a78cc..fa8151789 100644
--- a/tests/unit/amlb/datasets/file/test_file_dataloader.py
+++ b/tests/unit/amlb/datasets/file/test_file_dataloader.py
@@ -65,7 +65,8 @@ def test_load_binary_task_arff(file_loader):
 def _assert_kc2_features(dataset, definition):
     assert len(dataset.features) == 22
     assert len(dataset.predictors) == 21
-    _assert_target(dataset.target, name=definition.target, values=["no", "yes"])
+    target_values = ["no", "yes"]
+    _assert_target(dataset.target, name=definition.target, values=target_values)
 
     assert all([p.data_type in ['int', 'float'] for p in dataset.predictors])
     assert all([p.values is None for p in dataset.predictors])
@@ -78,6 +79,10 @@ def _assert_kc2_features(dataset, definition):
     assert dataset.train.X.dtypes.filter(items=ints).apply(lambda dt: pd.api.types.is_integer_dtype(dt)).all()
     assert pd.api.types.is_categorical_dtype(dataset.train.y.dtypes.iloc[0])
 
+    normalize = dataset.target.normalize
+    assert list(normalize(dataset.train.y.squeeze().unique())) == list(normalize(dataset.test.y.squeeze().unique())) == target_values
+    assert list(np.unique(dataset.train.y_enc)) == list(np.unique(dataset.test.y_enc)) == [0, 1]
+
 
 @pytest.mark.use_disk
 def test_load_multiclass_task_csv(file_loader):
@@ -94,6 +99,33 @@ def test_load_multiclass_task_csv(file_loader):
     _assert_iris_features(ds, ds_def)
 
 
+@pytest.mark.use_disk
+def test_load_multiclass_task_with_num_target_no_type_csv(file_loader):
+    ds_def = ns(
+        train=os.path.join(res, "iris_num_train.csv"),
+        test=os.path.join(res, "iris_num_test.csv"),
+        target="class"
+    )
+    ds = file_loader.load(ds_def)
+    assert ds.type is DatasetType.regression, "file loader should detect num target as regression by default"
+
+
+@pytest.mark.use_disk
+def test_load_multiclass_task_with_num_target_csv(file_loader):
+    ds_def = ns(
+        train=os.path.join(res, "iris_num_train.csv"),
+        test=os.path.join(res, "iris_num_test.csv"),
+        target="class",
+        type="multiclass"
+    )
+    ds = file_loader.load(ds_def)
+    assert ds.type is DatasetType.multiclass
+    _assert_X_y_types(ds.train)
+    _assert_data_consistency(ds)
+    _assert_data_paths(ds, ds_def)
+    _assert_iris_features(ds, ds_def, num_target=True)
+
+
 @pytest.mark.use_disk
 def test_load_multiclass_task_arff(file_loader):
     ds_def = ns(
@@ -109,10 +141,11 @@ def test_load_multiclass_task_arff(file_loader):
     _assert_iris_features(ds, ds_def)
 
 
-def _assert_iris_features(dataset, definition):
+def _assert_iris_features(dataset, definition, num_target=False):
     assert len(dataset.features) == 5
     assert len(dataset.predictors) == 4
-    _assert_target(dataset.target, name=definition.target, values=["Iris-setosa", "Iris-versicolor", "Iris-virginica"])  # values are case-sensitive when using file loader
+    target_values = ["1", "2", "3"] if num_target else ["iris-setosa", "iris-versicolor", "iris-virginica"]  # values are normalized
+    _assert_target(dataset.target, name=definition.target, values=target_values)
 
     assert all([p.data_type in ['int', 'float'] for p in dataset.predictors])
     assert all([p.values is None for p in dataset.predictors])
@@ -125,6 +158,10 @@ def _assert_iris_features(dataset, definition):
     assert dataset.train.X.dtypes.filter(items=ints).apply(lambda dt: pd.api.types.is_integer_dtype(dt)).all()
     assert pd.api.types.is_categorical_dtype(dataset.train.y.dtypes.iloc[0])
 
+    normalize = dataset.target.normalize
+    assert list(normalize(dataset.train.y.squeeze().unique())) == list(normalize(dataset.test.y.squeeze().unique())) == target_values
+    assert list(np.unique(dataset.train.y_enc)) == list(np.unique(dataset.test.y_enc)) == [0, 1, 2]
+
 
 @pytest.mark.use_disk
 def test_load_regression_task_csv(file_loader):
@@ -178,6 +215,9 @@ def _assert_cholesterol_features(dataset, definition, fmt):
     assert dataset.train.X.dtypes.filter(items=categoricals).apply(lambda dt: pd.api.types.is_categorical_dtype(dt)).all()
     assert pd.api.types.is_float_dtype(dataset.train.y.dtypes.iloc[0])
 
+    assert np.array_equal(dataset.train.y_enc, dataset.train.y.squeeze().to_numpy()), "no encoding should have been applied on regression target"
+    assert np.array_equal(dataset.test.y_enc, dataset.test.y.squeeze().to_numpy()), "no encoding should have been applied on regression target"
+
 
 def _assert_target(target, name, values=None):
     assert target.name == name
diff --git a/tests/unit/amlb/datasets/openml/test_openml_dataloader.py b/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
index 8b1efbb35..3c496d462 100644
--- a/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
+++ b/tests/unit/amlb/datasets/openml/test_openml_dataloader.py
@@ -113,8 +113,7 @@ def _assert_cholesterol_features(dataset):
 
     assert dataset.train.X.dtypes.filter(items=numericals).apply(lambda dt: pd.api.types.is_numeric_dtype(dt)).all()
     assert dataset.train.X.dtypes.filter(items=categoricals).apply(lambda dt: pd.api.types.is_categorical_dtype(dt)).all()
-    assert len(dataset.train.X.select_dtypes(include=['float']).columns) == 2
-    assert len(dataset.train.X.select_dtypes(include=['uint8']).columns) == 4
+    assert len(dataset.train.X.select_dtypes(include=['float']).columns) == 6
     assert pd.api.types.is_float_dtype(dataset.train.y.dtypes.iloc[0])
 
 
diff --git a/tests/unit/amlb/job/test_MultiThreadingJobRunner.py b/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
index 543c2285c..7510f1906 100644
--- a/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
+++ b/tests/unit/amlb/job/test_MultiThreadingJobRunner.py
@@ -124,6 +124,7 @@ def test_reschedule_job_enforce_job_priority():
 
     runner = MultiThreadingJobRunner(jobs, parallel_jobs=3, delay_secs=0.2,
                                      queueing_strategy=MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority)
+
     def _run(self, mock, ori):
         if mock.call_count < 3:
             self.reschedule()
@@ -228,9 +229,9 @@ def _on_state(self, state, ori=None):
 
     rescheduled_jobs = [j for i, j in enumerate(jobs) if i % 5 == 0]
     for job in rescheduled_jobs:
-        job._run = ft.partialmethod(_run, ori=job._run).__get__(job)
+        job._run = ft.partialmethod(_run, ori=job._run).__get__(job, None)
     for job in jobs:
-        job._on_state = ft.partialmethod(_on_state, ori=job._on_state).__get__(job)
+        job._on_state = ft.partialmethod(_on_state, ori=job._on_state).__get__(job, None)
 
     runner = MultiThreadingJobRunner(jobs, parallel_jobs=50, delay_secs=0.01,
                                      queueing_strategy=MultiThreadingJobRunner.QueueingStrategy.enforce_job_priority)
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json
new file mode 100644
index 000000000..4ad103514
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 0,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575752,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/0/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/0/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json
new file mode 100644
index 000000000..95be8717b
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 1,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575753,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/1/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/1/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json
new file mode 100644
index 000000000..b18f172c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 2,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575754,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/2/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/2/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json
new file mode 100644
index 000000000..e9ebbfc65
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 3,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575755,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2465,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/3/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/3/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json
new file mode 100644
index 000000000..06ebf6b37
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 4,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575756,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/4/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/4/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json
new file mode 100644
index 000000000..caf373827
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 5,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575757,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2464,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/5/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/5/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json
new file mode 100644
index 000000000..94e735223
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 6,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575758,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2461,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/6/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/6/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json
new file mode 100644
index 000000000..83af8f9bd
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 7,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575759,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2463,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/7/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/7/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json
new file mode 100644
index 000000000..ee7d85454
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 8,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575760,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2462,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/8/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/8/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json b/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json
new file mode 100644
index 000000000..16dd4df8c
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json
@@ -0,0 +1,64 @@
+{
+    "framework": "constantpredictor",
+    "framework_params": {},
+    "framework_version": "1.2.2",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 9,
+    "metric": "logloss",
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "seed": 1362575761,
+    "job_timeout_seconds": 7200,
+    "max_runtime_seconds": 3600,
+    "cores": 8,
+    "max_mem_size_mb": 2460,
+    "min_vol_size_mb": 16384,
+    "input_dir": "/Users/pietergijsbers/.openml",
+    "output_dir": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757",
+    "output_predictions_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/9/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py constantpredictor openml/t/1196 1h8c --openml-test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_on_upgrade_core",
+        "commit": "51e692e76e4fa6b6981cccdeee7769f2bd8eded1",
+        "tags": [],
+        "status": [
+            "## oml_on_upgrade_core...origin/oml_on_upgrade_core [ahead 1]",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/0/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/1/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/2/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/3/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/4/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/5/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/6/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/7/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/8/predictions.csv",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/metadata.json",
+            "D  tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv",
+            "?? catboost_1685648343.1131341/",
+            "?? catboost_1685648720.524352/",
+            "?? tests/unit/amlb/uploads/resources/iris_constant/",
+            "?? testvenv/",
+            "?? tttest.py"
+        ]
+    },
+    "ext": {},
+    "type_": "multiclass",
+    "output_metadata_file": "/Users/pietergijsbers/repositories/automlbenchmark/results/constantpredictor.openml_t_1196.1h8c.local.20230609T141757/predictions/iris/9/metadata.json"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv b/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv
new file mode 100644
index 000000000..3f87d18b3
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_constant/9/predictions.csv
@@ -0,0 +1,16 @@
+iris-setosa,iris-versicolor,iris-virginica,predictions,truth
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-setosa
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-versicolor
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
+0.3333333333333333,0.3333333333333333,0.3333333333333333,Iris-setosa,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json
new file mode 100644
index 000000000..7711c2c73
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/0/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 0,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415600,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22324,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/0/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/0/metadata.json",
+    "result_token": "0e96514e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp6q9_eifx",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv
new file mode 100644
index 000000000..26e5202af
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/0/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9946953997563966,0.0053046002344551485,9.148348333343108E-12,Iris-setosa,Iris-setosa
+0.9985317282709597,0.0014682717290367799,3.6521815928739156E-15,Iris-setosa,Iris-setosa
+0.9933721446687596,0.006627855330845209,3.952500544573547E-13,Iris-setosa,Iris-setosa
+0.9829199191390252,0.017080080834738047,2.623664472860213E-11,Iris-setosa,Iris-setosa
+0.9975765427390224,0.00242345726092685,5.062201017884491E-14,Iris-setosa,Iris-setosa
+0.0044465533046761044,0.9892091660989077,0.006344280596416092,Iris-versicolor,Iris-versicolor
+3.514690516726789E-5,0.9714925562042298,0.028472296890602915,Iris-versicolor,Iris-versicolor
+0.0010697343341420122,0.992866262840173,0.006064002825685055,Iris-versicolor,Iris-versicolor
+2.1266769099902806E-4,0.9986645376592181,0.0011227946497828257,Iris-versicolor,Iris-versicolor
+0.001757984985526363,0.9954098065054922,0.002832208508981477,Iris-versicolor,Iris-versicolor
+5.742239231322204E-6,0.33211727254434054,0.6678769852164281,Iris-virginica,Iris-virginica
+4.121706214335948E-7,0.031571169758323414,0.9684284180710553,Iris-virginica,Iris-virginica
+4.066267276715623E-9,3.412176268706058E-4,0.9996587783068621,Iris-virginica,Iris-virginica
+2.9567515352594244E-6,0.07228280087561359,0.927714242372851,Iris-virginica,Iris-virginica
+2.3982340907419163E-7,0.042374076481183916,0.9576256836954069,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json
new file mode 100644
index 000000000..88cf740e6
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/1/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 1,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415601,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/1/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/1/metadata.json",
+    "result_token": "3538d1b4-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpzzp9l29e",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv
new file mode 100644
index 000000000..1837d94ea
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/1/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9977407400831685,0.00225925991682751,4.015165004779329E-15,Iris-setosa,Iris-setosa
+0.9998127799097366,1.872200902633975E-4,4.4069911448376E-17,Iris-setosa,Iris-setosa
+0.9991029079294788,8.970920705179842E-4,3.36945787249407E-15,Iris-setosa,Iris-setosa
+0.9988618905736355,0.0011381094263636813,9.755423863959563E-16,Iris-setosa,Iris-setosa
+0.9991718877459559,8.281122540317876E-4,1.2331412068001223E-14,Iris-setosa,Iris-setosa
+1.4444276601415674E-5,0.9586804544702732,0.04130510125312546,Iris-versicolor,Iris-versicolor
+0.0013080421910475607,0.9914876185496758,0.0072043392592766094,Iris-versicolor,Iris-versicolor
+9.165272335157342E-5,0.9926865448386248,0.007221802438023577,Iris-versicolor,Iris-versicolor
+4.000656682613745E-5,0.9403412635245068,0.05961872990866716,Iris-versicolor,Iris-versicolor
+4.556745505745815E-4,0.9862291776381161,0.013315147811309238,Iris-versicolor,Iris-versicolor
+2.823761394672141E-5,0.1259734017180567,0.8739983606679965,Iris-virginica,Iris-virginica
+6.633076885499847E-9,3.6946619151488527E-4,0.9996305271754083,Iris-virginica,Iris-virginica
+1.124808863216433E-8,0.005488528420360704,0.9945114603315506,Iris-virginica,Iris-virginica
+4.547823734480297E-8,0.0026021657037141105,0.9973977888180486,Iris-virginica,Iris-virginica
+3.4159926848071144E-8,0.012462663957173407,0.9875373018828998,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json
new file mode 100644
index 000000000..1932fecbe
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/2/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 2,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415602,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/2/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/2/metadata.json",
+    "result_token": "5bbdfe5e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmppxvl4v2c",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv
new file mode 100644
index 000000000..f700a8d34
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/2/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.999643185421844,3.5681457815609174E-4,2.230159041262659E-17,Iris-setosa,Iris-setosa
+0.7411777171360305,0.2588222828628354,1.1341170862473228E-12,Iris-setosa,Iris-setosa
+0.9894857732646907,0.010514226735307778,1.4471496430130413E-15,Iris-setosa,Iris-setosa
+0.9995220482574042,4.7795174259571756E-4,3.52685599190754E-18,Iris-setosa,Iris-setosa
+0.9988481135869727,0.001151886413027263,3.987099704316224E-17,Iris-setosa,Iris-setosa
+0.0013884526297230128,0.9940428944867407,0.004568652883536275,Iris-versicolor,Iris-versicolor
+0.005953079691957121,0.9940145236999208,3.239660812213889E-5,Iris-versicolor,Iris-versicolor
+0.010116816444659752,0.9893970388624135,4.8614469292683455E-4,Iris-versicolor,Iris-versicolor
+7.23459628127077E-4,0.99243180940297,0.006844730968902987,Iris-versicolor,Iris-versicolor
+3.3659932451520145E-6,0.5553095425683089,0.4446870914384459,Iris-versicolor,Iris-versicolor
+1.2929154379113906E-12,7.240147885676274E-5,0.9999275985198504,Iris-virginica,Iris-virginica
+5.618995240209567E-8,0.002834747714656968,0.9971651960953906,Iris-virginica,Iris-virginica
+1.0168116273950983E-8,0.004773137506941937,0.9952268523249418,Iris-virginica,Iris-virginica
+1.1743944498400225E-12,3.7400521135450664E-5,0.9999625994776901,Iris-virginica,Iris-virginica
+3.555096876025579E-10,1.2269388421367656E-4,0.9998773057602767,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json
new file mode 100644
index 000000000..3ab354b47
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/3/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 3,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415603,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/3/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/3/metadata.json",
+    "result_token": "842080a6-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpju0mvnpz",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv
new file mode 100644
index 000000000..7888779c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/3/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9957981631401925,0.004201836859803345,4.248771336417623E-15,Iris-setosa,Iris-setosa
+0.9999242893981092,7.57106018908299E-5,7.555452439324001E-18,Iris-setosa,Iris-setosa
+0.9928309264483719,0.0071690735515837324,4.435073927352755E-14,Iris-setosa,Iris-setosa
+0.9981107676724997,0.0018892323274941557,6.2433444463775446E-15,Iris-setosa,Iris-setosa
+0.9992575208258436,7.424791741562481E-4,3.2743342900204155E-16,Iris-setosa,Iris-setosa
+0.00617078018723429,0.9937995194498253,2.9700362940417253E-5,Iris-versicolor,Iris-versicolor
+0.001856136695353521,0.9827598819837347,0.015383981320911793,Iris-versicolor,Iris-versicolor
+0.046721989468264224,0.9532569491771904,2.106135454538263E-5,Iris-versicolor,Iris-versicolor
+0.0023199782194149123,0.9971359443070136,5.440774735713279E-4,Iris-versicolor,Iris-versicolor
+1.2065477934082464E-5,0.4747201530513801,0.5252677814706859,Iris-virginica,Iris-versicolor
+1.1586735176370364E-9,4.4327508426450854E-4,0.9995567237570618,Iris-virginica,Iris-virginica
+8.213607614325511E-13,7.434115656964782E-5,0.9999256588426091,Iris-virginica,Iris-virginica
+6.260997775752786E-9,0.0015969357139136785,0.9984030580250886,Iris-virginica,Iris-virginica
+2.302723428633504E-10,1.6320985908881373E-5,0.9999836787838188,Iris-virginica,Iris-virginica
+5.077776188678325E-10,0.0022517395458113206,0.9977482599464109,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json
new file mode 100644
index 000000000..b31af78f4
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/4/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 4,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415604,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/4/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/4/metadata.json",
+    "result_token": "ae4c3d3e-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpdd0mphh0",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv
new file mode 100644
index 000000000..ebb85511f
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/4/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9995671767145491,4.3282328545065474E-4,2.910634990710733E-16,Iris-setosa,Iris-setosa
+0.9982697586656699,0.0017302413343282233,1.835406762143692E-15,Iris-setosa,Iris-setosa
+0.9989863122091491,0.0010136877908451531,5.704916978743878E-15,Iris-setosa,Iris-setosa
+0.9996616547927483,3.3834520725136086E-4,3.9294020366326627E-16,Iris-setosa,Iris-setosa
+0.9998864533391787,1.1354666082124777E-4,6.7435833537977226E-18,Iris-setosa,Iris-setosa
+0.004219356285716925,0.9920461471456139,0.003734496568669032,Iris-versicolor,Iris-versicolor
+6.839164491040382E-5,0.9945115713106095,0.005420037044480197,Iris-versicolor,Iris-versicolor
+0.0017683621545000424,0.9872280359587889,0.011003601886711033,Iris-versicolor,Iris-versicolor
+1.7369587491859734E-4,0.996893550822284,0.0029327533027972675,Iris-versicolor,Iris-versicolor
+4.354506494472319E-4,0.9882308872752477,0.011333662075305109,Iris-versicolor,Iris-versicolor
+2.697533881630151E-10,1.8877887113828945E-4,0.9998112208591085,Iris-virginica,Iris-virginica
+1.1833692210982864E-5,0.3136286688694338,0.6863594974383552,Iris-virginica,Iris-virginica
+3.860662541464733E-6,0.10171920502722513,0.8982769343102335,Iris-virginica,Iris-virginica
+2.6305660152696093E-7,0.003017310004465517,0.996982426938933,Iris-virginica,Iris-virginica
+3.5401683179702347E-7,0.020439351531391092,0.9795602944517771,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json
new file mode 100644
index 000000000..e0acd031b
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/5/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 5,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415605,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22328,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/5/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/5/metadata.json",
+    "result_token": "d6bb539a-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp6cunar4k",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv
new file mode 100644
index 000000000..a7164f84c
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/5/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9981088020680684,0.0018911979257595212,6.1720155217412105E-12,Iris-setosa,Iris-setosa
+0.9975129251674513,0.002487074799287478,3.3261253297507724E-11,Iris-setosa,Iris-setosa
+0.9872452587941326,0.012754741121718594,8.414891781400091E-11,Iris-setosa,Iris-setosa
+0.9865050960521858,0.013494903905854394,4.1959818974613784E-11,Iris-setosa,Iris-setosa
+0.9936909152559753,0.006309084608078586,1.3594606117458185E-10,Iris-setosa,Iris-setosa
+1.885964249056789E-4,0.3237625504836179,0.6760488530914763,Iris-virginica,Iris-versicolor
+0.006735391686925818,0.9524508509807971,0.04081375733227718,Iris-versicolor,Iris-versicolor
+5.378093523395172E-5,0.5228690867359151,0.4770771323288508,Iris-versicolor,Iris-versicolor
+3.5547125401524113E-4,0.8535160413485426,0.1461284873974421,Iris-versicolor,Iris-versicolor
+0.001280556645189532,0.9617810748971887,0.036938368457621766,Iris-versicolor,Iris-versicolor
+7.107012029851008E-5,0.29850295172759583,0.7014259781521057,Iris-virginica,Iris-virginica
+3.3942170781234547E-7,0.00447238089710375,0.9955272796811885,Iris-virginica,Iris-virginica
+1.895791713128563E-7,0.006446713099335271,0.9935530973214934,Iris-virginica,Iris-virginica
+4.3570097930097914E-4,0.3231055513427694,0.6764587476779297,Iris-virginica,Iris-virginica
+9.267662568271636E-5,0.05987127606163897,0.9400360473126782,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json
new file mode 100644
index 000000000..37ac517da
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/6/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 6,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415606,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/6/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/6/metadata.json",
+    "result_token": "fe15ff3a-445d-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmp_hfr1i3w",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv
new file mode 100644
index 000000000..3c1517540
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/6/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9986825173040808,0.0013174826959192415,2.8050853758906494E-17,Iris-setosa,Iris-setosa
+0.9998816031149136,1.1839688508629242E-4,9.723844873423558E-20,Iris-setosa,Iris-setosa
+0.9999856048453758,1.4395154624238886E-5,3.7272699996634135E-22,Iris-setosa,Iris-setosa
+0.9958896321731253,0.004110367826874745,9.1106915338899E-18,Iris-setosa,Iris-setosa
+0.9994195014384868,5.804985615131422E-4,8.352468371125052E-19,Iris-setosa,Iris-setosa
+1.0187624255752032E-4,0.9808217812619279,0.019076342495514632,Iris-versicolor,Iris-versicolor
+0.005455799247359012,0.9941389680709293,4.052326817118014E-4,Iris-versicolor,Iris-versicolor
+0.006332155970539919,0.9936285090877568,3.933494170343988E-5,Iris-versicolor,Iris-versicolor
+2.164701150526862E-4,0.9920704142697138,0.007713115615233376,Iris-versicolor,Iris-versicolor
+0.0012505917761682515,0.9987282670228536,2.1141200978234624E-5,Iris-versicolor,Iris-versicolor
+2.2087333569148024E-7,0.03154911382942394,0.9684506652972403,Iris-virginica,Iris-virginica
+8.458349099345969E-8,0.007958875829817774,0.9920410395866912,Iris-virginica,Iris-virginica
+3.5546202750827647E-12,7.378039045762072E-4,0.9992621960918691,Iris-virginica,Iris-virginica
+3.148087616934334E-18,1.0418835083791423E-7,0.9999998958116492,Iris-virginica,Iris-virginica
+3.935392414196986E-11,0.0011083045592472595,0.9988916954013989,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json
new file mode 100644
index 000000000..d18694044
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/7/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 7,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415607,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22329,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/7/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/7/metadata.json",
+    "result_token": "2416e3f2-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpa865dt_n",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv
new file mode 100644
index 000000000..f04ea4aae
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/7/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9990348975998995,9.651024001005388E-4,2.305810635776546E-17,Iris-setosa,Iris-setosa
+0.9994251478468725,5.748521531275645E-4,2.741195791809452E-18,Iris-setosa,Iris-setosa
+0.9974698813122362,0.0025301186877638043,1.4929300527492385E-17,Iris-setosa,Iris-setosa
+0.9980293757904823,0.001970624209517628,5.742383231957409E-18,Iris-setosa,Iris-setosa
+0.9989317358150057,0.0010682641849941218,3.939261654207119E-17,Iris-setosa,Iris-setosa
+3.700937950335641E-5,0.9867216516835606,0.013241338936936058,Iris-versicolor,Iris-versicolor
+0.0013687044804676057,0.9985112580034591,1.2003751607327173E-4,Iris-versicolor,Iris-versicolor
+0.006995911111617461,0.9682817728451619,0.02472231604322051,Iris-versicolor,Iris-versicolor
+1.3802427438593855E-4,0.9992264464192203,6.355293063938353E-4,Iris-versicolor,Iris-versicolor
+0.04709941690670438,0.9528951693622032,5.413731092497622E-6,Iris-versicolor,Iris-versicolor
+4.688538283391994E-11,5.7853607133277914E-5,0.9999421463459814,Iris-virginica,Iris-virginica
+1.6509551800831845E-5,0.7396217633784552,0.260361727069744,Iris-versicolor,Iris-virginica
+6.724194037868728E-4,0.3117625931928038,0.6875649874034093,Iris-virginica,Iris-virginica
+5.408383401537574E-10,4.153769951126476E-4,0.9995846224640491,Iris-virginica,Iris-virginica
+5.395915212183842E-11,4.1105559071968754E-4,0.9995889443553212,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json
new file mode 100644
index 000000000..0e52712c9
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/8/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 8,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415608,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22330,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/8/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/8/metadata.json",
+    "result_token": "4be39e52-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpgdovn6fi",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv
new file mode 100644
index 000000000..621acadad
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/8/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9922545822525884,0.007745417747410964,7.085353966139096E-16,Iris-setosa,Iris-setosa
+0.9963262691619229,0.0036737308380769234,1.5463726893802133E-16,Iris-setosa,Iris-setosa
+0.9997012711968896,2.9872880311049033E-4,1.7177502211229142E-17,Iris-setosa,Iris-setosa
+0.9982734171658221,0.0017265828341778042,1.953013147945192E-16,Iris-setosa,Iris-setosa
+0.9987322082865054,0.0012677917134945812,8.816555725807466E-18,Iris-setosa,Iris-setosa
+9.23865315845644E-4,0.9990174169021824,5.871778197195178E-5,Iris-versicolor,Iris-versicolor
+1.250278151239352E-4,0.9642979508172816,0.03557702136759442,Iris-versicolor,Iris-versicolor
+0.0013413343296529907,0.9299740756890498,0.06868458998129721,Iris-versicolor,Iris-versicolor
+8.35292503326207E-5,0.28583411631968514,0.7140823544299824,Iris-virginica,Iris-versicolor
+0.004834483157012863,0.9883085749200913,0.006856941922895849,Iris-versicolor,Iris-versicolor
+2.9816510566649863E-10,4.55874025606045E-5,0.9999544122992742,Iris-virginica,Iris-virginica
+1.2716794291346635E-5,0.229055275940032,0.7709320072656767,Iris-virginica,Iris-virginica
+2.440835419324771E-8,0.22493379307168823,0.7750661825199575,Iris-virginica,Iris-virginica
+6.1155599921910175E-9,0.0015434973500508815,0.9984564965343892,Iris-virginica,Iris-virginica
+2.1813458093942627E-11,1.0099649947803468E-4,0.9998990034787085,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json b/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json
new file mode 100644
index 000000000..fd5c3ffd8
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/9/metadata.json
@@ -0,0 +1,51 @@
+{
+    "framework": "H2OAutoML",
+    "framework_params": {},
+    "framework_version": "stable",
+    "type": "classification",
+    "name": "iris",
+    "openml_task_id": 1196,
+    "test_server": true,
+    "fold": 9,
+    "metrics": [
+        "logloss",
+        "acc",
+        "balacc"
+    ],
+    "metric": "logloss",
+    "seed": 413415609,
+    "max_runtime_seconds": 60,
+    "cores": 4,
+    "max_mem_size_mb": 22330,
+    "min_vol_size_mb": -1,
+    "input_dir": "/root/.openml/cache",
+    "output_dir": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155",
+    "output_predictions_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/9/predictions.csv",
+    "tag": null,
+    "command": "runbenchmark.py h2oautoml openml/t/1196 test --test-server",
+    "git_info": {
+        "repo": "https://github.com/openml/automlbenchmark.git",
+        "branch": "oml_upload",
+        "commit": "482b6e6b61258f3ace36d91a4a72756ac45b61e5",
+        "tags": [],
+        "status": [
+            "## oml_upload...origin/oml_upload",
+            " M resources/constraints.yaml"
+        ]
+    },
+    "ext": {
+        "monitoring": {
+            "frequency_seconds": 120,
+            "statistics": [
+                "cpu",
+                "memory",
+                "volume"
+            ],
+            "verbosity": 0
+        }
+    },
+    "output_metadata_file": "/automlbenchmark/results/h2oautoml.openml_t_1196.test.local.20201222T135155/predictions/iris/9/metadata.json",
+    "result_token": "7468049e-445e-11eb-baf3-0242ac110002",
+    "result_dir": "/tmp/tmpyme85l1y",
+    "version": "3.32.0.2"
+}
\ No newline at end of file
diff --git a/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv b/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv
new file mode 100644
index 000000000..d71e4bc7e
--- /dev/null
+++ b/tests/unit/amlb/uploads/resources/iris_h2o/9/predictions.csv
@@ -0,0 +1,16 @@
+Iris-setosa,Iris-versicolor,Iris-virginica,predictions,truth
+0.9994560256672776,5.439593975257684E-4,1.4935196648718606E-8,Iris-setosa,Iris-setosa
+0.9991506482838198,8.493452971975329E-4,6.4189825553693556E-9,Iris-setosa,Iris-setosa
+0.9967761624035782,0.0032238257275304743,1.1868891386506383E-8,Iris-setosa,Iris-setosa
+0.9985515492213874,0.001448441407003948,9.371608537457098E-9,Iris-setosa,Iris-setosa
+0.9988119885708648,0.0011880061718968458,5.257238382408336E-9,Iris-setosa,Iris-setosa
+0.01529292273184757,0.6326831426347472,0.35202393463340526,Iris-versicolor,Iris-versicolor
+0.0028013111855947344,0.5669742812230935,0.43022440759131186,Iris-versicolor,Iris-versicolor
+0.005414416565513969,0.9799766510832394,0.014608932351246532,Iris-versicolor,Iris-versicolor
+0.0035872791673992248,0.9478077493917355,0.048604971440865216,Iris-versicolor,Iris-versicolor
+0.0045389422457486435,0.9914975512751514,0.003963506479099829,Iris-versicolor,Iris-versicolor
+3.750897792875574E-5,0.17921483171391367,0.8207476593081576,Iris-virginica,Iris-virginica
+3.166596517299125E-5,0.31433974404012566,0.6856285899947012,Iris-virginica,Iris-virginica
+4.694421902512631E-7,0.011834435964019752,0.98816509459379,Iris-virginica,Iris-virginica
+9.957182955200623E-6,0.09057334922636441,0.9094166935906803,Iris-virginica,Iris-virginica
+4.998346379369319E-6,0.207413278285669,0.7925817233679515,Iris-virginica,Iris-virginica
diff --git a/tests/unit/amlb/uploads/test_file_loading.py b/tests/unit/amlb/uploads/test_file_loading.py
new file mode 100644
index 000000000..76b47a067
--- /dev/null
+++ b/tests/unit/amlb/uploads/test_file_loading.py
@@ -0,0 +1,103 @@
+import os
+from collections import OrderedDict
+import pathlib
+
+import openml
+import pandas as pd
+import pytest
+
+from amlb.uploads import _load_predictions, _load_fold, _get_flow, _load_task_data, \
+    _extract_and_format_hyperparameter_configuration, _upload_results
+
+here = pathlib.Path(__file__).parent.absolute()
+resources = here / 'resources'
+
+iris_constant = resources / "iris_constant"
+iris_h2o = resources / "iris_h2o"
+
+
+@pytest.fixture(scope="module")
+def with_oml_test_server():
+    openml.config.start_using_configuration_for_example()
+    yield
+    openml.config.stop_using_configuration_for_example()
+
+
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__load_fold_loads_prediction_file(with_oml_test_server):
+    iris_task = openml.tasks.get_task(1196)
+    predictions = _load_fold(iris_constant, 0, iris_task)
+    assert isinstance(predictions, pd.DataFrame)
+    assert predictions.shape == (15, 8)
+    assert set(predictions.columns) == {"repeat", "fold", "index", "iris-setosa",
+                                        "iris-versicolor", "iris-virginica",
+                                        "predictions", "truth"}
+
+
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__load_fold_attaches_correct_index_information(with_oml_test_server):
+    iris_task = openml.tasks.get_task(1196)
+    predictions = _load_fold(iris_constant, 0, iris_task)
+    assert list(predictions["index"]) == [43, 14, 37, 23, 10, 99, 87, 97, 62, 92, 119,
+                                          111, 144, 116, 125]
+    assert all(predictions["fold"] == 0)
+    predictions = _load_fold(iris_constant, 1, iris_task)
+    assert list(predictions["index"]) == [49, 15, 47, 0, 44, 76, 89, 58, 54, 53, 149,
+                                          114, 112, 115, 139]
+    assert all(predictions["fold"] == 1)
+
+
+@pytest.mark.use_disk
+def test__load_predictions_loads_all_files():
+    predictions = _load_predictions(iris_constant)
+    assert isinstance(predictions, pd.DataFrame)
+    assert predictions.shape == (150, 8)
+    assert set(predictions.columns) == {"repeat", "fold", "index", "iris-setosa",
+                                        "iris-versicolor", "iris-virginica",
+                                        "predictions", "truth"}
+    assert set(predictions["fold"]) == set(range(10))
+
+
+@pytest.mark.use_disk
+def test__extract_flow_hyperparameter_configuration():
+    metadata = _load_task_data(iris_constant)
+    flow = _get_flow(metadata, sync_with_server=False)
+    parameters = _extract_and_format_hyperparameter_configuration(metadata, flow)
+
+    expected_parameters = [
+        OrderedDict([("oml:name", "max_runtime_seconds"), ("oml:value", 3600),
+                     ("oml:component", flow.id)]),
+        OrderedDict([("oml:name", "max_mem_size_mb"), ("oml:value", 2465),
+                     ("oml:component", flow.id)]),
+        OrderedDict(
+            [("oml:name", "cores"), ("oml:value", 8), ("oml:component", flow.id)]),
+        OrderedDict([("oml:name", "seed"), ("oml:value", 1362575752),
+                     ("oml:component", flow.id)]),
+    ]
+    assert parameters == expected_parameters
+
+
+@pytest.mark.openml_upload
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__upload_results_constant(with_oml_test_server):
+    run = _upload_results(iris_constant)
+    assert isinstance(run.id, int)
+
+
+@pytest.mark.openml_upload
+@pytest.mark.use_web
+@pytest.mark.use_disk
+def test__upload_results_h2o(with_oml_test_server):
+    run = _upload_results(iris_h2o)
+    assert isinstance(run.id, int)
+
+
+@pytest.mark.skip(
+    "I can't think of a meaningful test that doesn't just verify that openml-python works as expected."
+)
+def test__get_flow(with_oml_test_server):
+    metadata = _load_task_data(iris_constant)
+    flow = _get_flow(metadata)