Repository: conbench/conbench (language: python)

Issue:
legacy package: Create separate package for legacy CLI and library
Breaking this out of #617 to be the first step: Duplicate the legacy code (`conbench` CLI, benchmark runner, `Conbench` class) into a new package. This does _not_ include removing that code from its current locations yet; that will happen separately after users have been switched over to the new legacy package.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/actions.yml b/.github/workflows/actions.yml
index c96aec662..871e4537a 100755
--- a/.github/workflows/actions.yml
+++ b/.github/workflows/actions.yml
@@ -54,7 +54,7 @@ jobs:
             -e ./benchadapt/python \
             -e ./benchconnect \
             -e ./benchrun/python \
-            -e .
+            -e ./legacy
       - name: Build docs, exiting non-zero on warnings (which can be very bad)
         run: make build-docs SPHINXOPTS='-W --keep-going'
 
@@ -178,7 +178,8 @@ jobs:
             -e './benchalerts[dev]' \
             -e ./benchadapt/python \
             -e ./benchconnect \
-            -e ./benchrun/python
+            -e ./benchrun/python \
+            -e './legacy[dev]'
       - name: Test benchclients
         run: |
           pytest -vv benchclients/python/tests
@@ -197,6 +198,9 @@ jobs:
       - name: Test benchconnect
         run: |
           pytest -vv benchconnect
+      - name: Test conbenchlegacy
+        run: |
+          pytest -vv legacy
 
   cb-on-minikube:
     runs-on: ubuntu-latest
diff --git a/.github/workflows/publish_docs.yml b/.github/workflows/publish_docs.yml
index 44041cdcf..0c19d15d9 100644
--- a/.github/workflows/publish_docs.yml
+++ b/.github/workflows/publish_docs.yml
@@ -23,7 +23,7 @@ jobs:
             -e ./benchadapt/python \
             -e ./benchconnect \
             -e ./benchrun/python \
-            -e .
+            -e ./legacy
       - name: Build docs, exiting non-zero on warnings (which can be very bad)
         run: make build-docs SPHINXOPTS='-W --keep-going'
       - name: Upload docs as an artifact
diff --git a/docs/api_docs/conbench.rst b/docs/api_docs/conbench.rst
deleted file mode 100644
index 53c9120dc..000000000
--- a/docs/api_docs/conbench.rst
+++ /dev/null
@@ -1,12 +0,0 @@
-``conbench`` python library API documentation
-=============================================
-
-Note: this is a legacy package, intended to be replaced by the other packages in this
-repo.
-
-``conbench.runner`` module
---------------------------
-
-.. automodule:: conbench.runner
-   :members:
-   :show-inheritance:
diff --git a/docs/api_docs/conbenchlegacy.rst b/docs/api_docs/conbenchlegacy.rst
new file mode 100644
index 000000000..e36cc18bf
--- /dev/null
+++ b/docs/api_docs/conbenchlegacy.rst
@@ -0,0 +1,12 @@
+``conbenchlegacy`` python library API documentation
+===================================================
+
+Note: this is a legacy package, intended to be replaced by the other packages in this
+repo.
+
+``conbenchlegacy.runner`` module
+--------------------------------
+
+.. automodule:: conbenchlegacy.runner
+   :members:
+   :show-inheritance:
diff --git a/docs/index.rst b/docs/index.rst
index dddcbfc19..b389be129 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -22,7 +22,7 @@
     api_docs/benchconnect
     pages/benchrun_readme
     api_docs/benchrun
-    api_docs/conbench
+    api_docs/conbenchlegacy
 
 .. include:: ../README.md
     :parser: myst_parser.sphinx_
diff --git a/legacy/README.md b/legacy/README.md
new file mode 100755
index 000000000..80688a270
--- /dev/null
+++ b/legacy/README.md
@@ -0,0 +1,858 @@
+# Conbench Legacy CLI and Library
+
+This library legacy elements designed for use on clients from the main webapp.
+This includes classes for defining, running, and posting benchmarks, as well as the CLI for running them from the command line.
+
+<br>
+
+
+## Installation
+
+All packages in this repo can be installed from PyPI. Each package uses
+[CalVer](https://calver.org/) for versioning. No stability is guaranteed between PyPI
+versions, so consider pinning packages to a specific version in your code.
+
+```bash
+pip install 'conbenchlegacy@git+https://github.com/conbench/conbench.git@main#subdirectory=legacy'
+```
+
+## Authoring benchmarks
+
+There are three main types of benchmarks: "simple benchmarks" that time the
+execution of a unit of work, "external benchmarks" that just record benchmark
+results that were obtained from some other benchmarking tool, and "case
+benchmarks" which benchmark a unit of work under different scenarios (cases).
+
+Included in this repository are contrived, minimal examples of these different
+kinds of benchmarks to be used as templates for benchmark authoring. These
+example benchmarks and their tests can be found here:
+
+
+* [_example_benchmarks.py](https://github.com/conbench/conbench/blob/main/legacy/tests/benchmark/_example_benchmarks.py)
+* [test_cli.py](https://github.com/conbench/conbench/blob/main/legacy/tests/benchmark/test_cli.py)
+* [test_runner.py](https://github.com/conbench/conbench/blob/main/legacy/tests/benchmark/test_runner.py)
+
+
+### Example simple benchmarks
+
+A "simple benchmark" runs and records the execution time of a unit of work.
+
+Implementation details: Note that this benchmark extends
+`conbenchlegacy.runner.Benchmark`, implements the minimum required `run()`
+method, and registers itself with the `@conbenchlegacy.runner.register_benchmark`
+decorator.
+
+```python
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class SimpleBenchmark(conbenchlegacy.runner.Benchmark):
+    name = "addition"
+
+    def run(self, **kwargs):
+        yield self.conbenchlegacy.benchmark(
+            self._get_benchmark_function(), self.name, options=kwargs
+        )
+
+    def _get_benchmark_function(self):
+        return lambda: 1 + 1
+```
+
+Successfully registered benchmarks appear in the `conbench --help` list.
+For this benchmark to appear, the file must match the following paterns (documented in the [`utils.py:register_benchmarks` function](https://github.com/conbench/conbench/blob/35fde9954e2b365e36472f16b46fe3067e9dc455/conbench/util.py#L104):
+* `benchmark*.py`
+* `*benchmark.py`
+* `*benchmarks.py`
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench --help
+Usage: conbench [OPTIONS] COMMAND [ARGS]...
+
+  Conbench: Language-independent Continuous Benchmarking (CB) Framework
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  addition            Run addition benchmark.
+  external            Run external benchmark.
+  external-r          Run external-r benchmark.
+  external-r-options  Run external-r-options benchmark.
+  list                List of benchmarks (for orchestration).
+  matrix              Run matrix benchmark(s).
+  version             Display Conbench version.
+```
+
+Benchmarks can be run from command line within the directory where the
+benchmarks are defined.
+
+Benchmark classes can also be imported and executed via the `run` method which
+accepts the same arguments that appear in the command line help.
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench addition --help
+
+Usage: conbench addition [OPTIONS]
+
+  Run addition benchmark.
+
+Options:
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --gc-collect BOOLEAN   [default: true]
+  --gc-disable BOOLEAN   [default: true]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+```
+
+Example command line execution:
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench addition
+
+Benchmark result:
+{
+    "batch_id": "c9db942c27db4359923eb08aa553beb7",
+    "run_id": "f6c7d0b3b3f146f9b1ad297fc6e5776b",
+    "timestamp": "2021-11-09T22:09:17.790397+00:00",
+    "context": {
+        "benchmark_language": "Python"
+    },
+    "github": {
+        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
+        "repository": "https://github.com/conbench/conbench"
+    },
+    "info": {
+        "benchmark_language_version": "Python 3.9.7"
+    },
+    "machine_info": {
+        "architecture_name": "arm64",
+        "cpu_core_count": "8",
+        "cpu_frequency_max_hz": "0",
+        "cpu_l1d_cache_bytes": "65536",
+        "cpu_l1i_cache_bytes": "131072",
+        "cpu_l2_cache_bytes": "4194304",
+        "cpu_l3_cache_bytes": "0",
+        "cpu_model_name": "Apple M1",
+        "cpu_thread_count": "8",
+        "gpu_count": "0",
+        "gpu_product_names": [],
+        "kernel_name": "20.6.0",
+        "memory_bytes": "17179869184",
+        "name": "diana",
+        "os_name": "macOS",
+        "os_version": "11.5.2"
+    },
+    "stats": {
+        "data": [
+            "0.000001"
+        ],
+        "iqr": "0.000000",
+        "iterations": 1,
+        "max": "0.000001",
+        "mean": "0.000001",
+        "median": "0.000001",
+        "min": "0.000001",
+        "q1": "0.000001",
+        "q3": "0.000001",
+        "stdev": 0,
+        "time_unit": "s",
+        "times": [],
+        "unit": "s"
+    },
+    "tags": {
+        "name": "addition"
+    }
+}
+```
+
+Example Python execution:
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ python
+>>> import json
+>>> import _example_benchmarks
+>>> benchmark = _example_benchmarks.SimpleBenchmark()
+>>> [(result, output)] = benchmark.run(iterations=10)
+>>> output
+2
+>>> print(json.dumps(result, indent=2))
+{
+  "run_id": "dfe3a816ca9e451a9da7d940a974cb95",
+  "batch_id": "0e869934b391424a8199c485dfbbc066",
+  "timestamp": "2021-11-09T22:11:25.262330+00:00",
+  "stats": {
+    "data": [
+      "0.000002",
+      "0.000001",
+      "0.000000",
+      "0.000001",
+      "0.000001",
+      "0.000001",
+      "0.000001",
+      "0.000000",
+      "0.000001",
+      "0.000001"
+    ],
+    "times": [],
+    "unit": "s",
+    "time_unit": "s",
+    "iterations": 10,
+    "mean": "0.000001",
+    "median": "0.000001",
+    "min": "0.000000",
+    "max": "0.000002",
+    "stdev": "0.000001",
+    "q1": "0.000001",
+    "q3": "0.000001",
+    "iqr": "0.000000"
+  },
+  "machine_info": {
+    "name": "diana",
+    "os_name": "macOS",
+    "os_version": "11.5.2",
+    "architecture_name": "arm64",
+    "kernel_name": "20.6.0",
+    "memory_bytes": "17179869184",
+    "cpu_model_name": "Apple M1",
+    "cpu_core_count": "8",
+    "cpu_thread_count": "8",
+    "cpu_l1d_cache_bytes": "65536",
+    "cpu_l1i_cache_bytes": "131072",
+    "cpu_l2_cache_bytes": "4194304",
+    "cpu_l3_cache_bytes": "0",
+    "cpu_frequency_max_hz": "0",
+    "gpu_count": "0",
+    "gpu_product_names": []
+  },
+  "context": {
+    "benchmark_language": "Python"
+  },
+  "info": {
+    "benchmark_language_version": "Python 3.9.7"
+  },
+  "tags": {
+    "name": "addition"
+  },
+  "github": {
+    "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
+    "repository": "https://github.com/conbench/conbench"
+  }
+}
+```
+
+By default, Conbench will try to publish your results to a Conbench server. If
+you don't have one running or are missing a `.conbench` credentials file, you'll
+see error messages like the following when you execute benchmarks.
+
+```
+POST http://localhost:5000/api/login/ failed
+{
+  "code": 400,
+  "description": {
+    "_errors": [
+      "Invalid email or password."
+    ]
+  },
+  "name": "Bad Request"
+}
+
+POST http://localhost:5000/api/benchmarks/ failed
+{
+  "code": 401,
+  "name": "Unauthorized"
+}
+```
+
+To publish your results to a Conbench server, place a `.conbench` file in the
+same directory as your benchmarks. The `cat` command below shows the contents
+of an example `.conbench` config file.
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ cat .conbench
+url: http://localhost:5000
+email: you@example.com
+password: conbench
+```
+
+If you don't yet have a Conbench server user account, you'll need to create one
+to publish results (registration key defaults to `conbench`).
+
+- http://localhost:5000/register/
+
+
+### Example external benchmarks
+
+An "external benchmark" records results that were obtained from some other
+benchmarking tool (like executing an R benchmark from command line, parsing
+the resulting JSON, and recording those results).
+
+Implementation details: Note that the following benchmark sets
+`external = True`, and calls `self.conbench.record()` rather than
+`self.conbench.benchmark()` as the example above does.
+
+```python
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that just records external results."""
+
+    external = True
+    name = "external"
+
+    def run(self, **kwargs):
+        # external results from an API call, command line execution, etc
+        result = {
+            "data": [100, 200, 300],
+            "unit": "i/s",
+            "times": [0.100, 0.200, 0.300],
+            "time_unit": "s",
+        }
+
+        context = {"benchmark_language": "C++"}
+        yield self.conbench.record(
+            result, self.name, context=context, options=kwargs, output=result
+        )
+```
+
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench external --help
+
+Usage: conbench external [OPTIONS]
+
+  Run external benchmark.
+
+Options:
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+```
+
+
+Note that the use of `--iterations=3` results in 3 runs of the benchmark, and
+the `mean`, `stdev`, etc calculated.
+
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench external --iterations=3
+
+Benchmark result:
+{
+    "run_id": "8058dde1491b49e5bd514646797c2a20",
+    "batch_id": "8058dde1491b49e5bd514646797c2a20",
+    "timestamp": "2021-06-21T22:16:54.786499+00:00",
+    "context": {
+        "benchmark_language": "C++"
+    },
+    "github": {
+        "commit": "58fb35dc593dca82c917cf18c1c65c059b9fb12c",
+        "repository": "https://github.com/conbench/conbench"
+    },
+    "info": {},
+    "machine_info": {
+        "architecture_name": "x86_64",
+        "cpu_core_count": "2",
+        "cpu_frequency_max_hz": "3500000000",
+        "cpu_l1d_cache_bytes": "32768",
+        "cpu_l1i_cache_bytes": "32768",
+        "cpu_l2_cache_bytes": "262144",
+        "cpu_l3_cache_bytes": "4194304",
+        "cpu_model_name": "Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz",
+        "cpu_thread_count": "4",
+        "kernel_name": "20.5.0",
+        "memory_bytes": "17179869184",
+        "name": "machine-abc",
+        "os_name": "macOS",
+        "os_version": "10.16"
+    },
+    "stats": {
+        "data": [
+            "100.000000",
+            "200.000000",
+            "300.000000"
+        ],
+        "iqr": "100.000000",
+        "iterations": 3,
+        "max": "300.000000",
+        "mean": "200.000000",
+        "median": "200.000000",
+        "min": "100.000000",
+        "q1": "150.000000",
+        "q3": "250.000000",
+        "stdev": "100.000000",
+        "time_unit": "s",
+        "times": [
+            "0.100000",
+            "0.200000",
+            "0.300000"
+        ],
+        "unit": "i/s"
+    },
+    "tags": {
+        "name": "external"
+    }
+}
+```
+
+### Example simple benchmarks executed on machine cluster instead of one machine
+
+If your benchmark is executed on a machine cluster instead of one machine, you
+can capture cluster info in the following manner.
+Note that a benchmark will have a continuous history on a specific cluster as long as cluster's `name` and `info` do not change.
+There is also an `optional_info` field for information that should not impact the cluster's hash (and thus disrupt the distribution history), but should still be recorded.
+```python
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class SimpleBenchmarkWithClusterInfo(conbenchlegacy.runner.Benchmark):
+    name = "product"
+
+    def run(self, **kwargs):
+        cluster_info = {
+            "name": "cluster 1",
+            "info": {"gpu": 1},
+            "optional_info": {"workers": 2},
+        }
+        yield self.conbench.benchmark(
+            self._get_benchmark_function(),
+            self.name,
+            cluster_info=cluster_info,
+            options=kwargs,
+        )
+
+    def _get_benchmark_function(self):
+        return lambda: 1 * 2
+```
+
+### Example case benchmarks
+
+A "case benchmark" is a either a "simple benchmark" or an "external benchmark"
+executed under various predefined scenarios (cases).
+
+Implementation details: Note that the following benchmark declares the valid
+combinations in `valid_cases`, which reads like a CSV (the first row contains
+the cases names).
+
+
+```python
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class CasesBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark with cases."""
+
+    name = "matrix"
+    valid_cases = (
+        ("rows", "columns"),
+        ("10", "10"),
+        ("2", "10"),
+        ("10", "2"),
+    )
+
+    def run(self, case=None, **kwargs):
+        for case in self.get_cases(case, kwargs):
+            rows, columns = case
+            tags = {"rows": rows, "columns": columns}
+            func = self._get_benchmark_function(rows, columns)
+            benchmark, output = self.conbench.benchmark(
+                func,
+                self.name,
+                tags=tags,
+                options=kwargs,
+            )
+            yield benchmark, output
+
+    def _get_benchmark_function(self, rows, columns):
+        return lambda: int(rows) * [int(columns) * [0]]
+```
+
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench matrix --help
+
+Usage: conbench matrix [OPTIONS]
+
+  Run matrix benchmark(s).
+
+  For each benchmark option, the first option value is the default.
+
+  Valid benchmark combinations:
+  --rows=10 --columns=10
+  --rows=2 --columns=10
+  --rows=10 --columns=2
+
+  To run all combinations:
+  $ conbench matrix --all=true
+
+Options:
+  --rows [10|2]
+  --columns [10|2]
+  --all BOOLEAN          [default: false]
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --gc-collect BOOLEAN   [default: true]
+  --gc-disable BOOLEAN   [default: true]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+    """
+```
+
+
+Note that the use of `--all=true` results in 3 benchmark results, one for each
+case (`10 x 10`, `2, x 10`, and `10, x 2`).
+
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench matrix --all=true
+
+Benchmark result:
+{
+    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
+    "run_id": "48acd853b8294df9a1f5457f192456f3",
+    "timestamp": "2021-11-09T22:15:23.501923+00:00",
+    "context": {
+        "benchmark_language": "Python"
+    },
+    "github": {
+        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
+        "repository": "https://github.com/conbench/conbench"
+    },
+    "info": {
+        "benchmark_language_version": "Python 3.9.7"
+    },
+    "machine_info": {
+        "architecture_name": "arm64",
+        "cpu_core_count": "8",
+        "cpu_frequency_max_hz": "0",
+        "cpu_l1d_cache_bytes": "65536",
+        "cpu_l1i_cache_bytes": "131072",
+        "cpu_l2_cache_bytes": "4194304",
+        "cpu_l3_cache_bytes": "0",
+        "cpu_model_name": "Apple M1",
+        "cpu_thread_count": "8",
+        "gpu_count": "0",
+        "gpu_product_names": [],
+        "kernel_name": "20.6.0",
+        "memory_bytes": "17179869184",
+        "name": "diana",
+        "os_name": "macOS",
+        "os_version": "11.5.2"
+    },
+    "run_id": "48acd853b8294df9a1f5457f192456f3",
+    "stats": {
+        "data": [
+            "0.000004"
+        ],
+        "iqr": "0.000000",
+        "iterations": 1,
+        "max": "0.000004",
+        "mean": "0.000004",
+        "median": "0.000004",
+        "min": "0.000004",
+        "q1": "0.000004",
+        "q3": "0.000004",
+        "stdev": 0,
+        "time_unit": "s",
+        "times": [],
+        "unit": "s"
+    },
+    "tags": {
+        "columns": "10",
+        "name": "matrix",
+        "rows": "10"
+    },
+    "timestamp": "2021-11-09T22:15:23.397819+00:00"
+}
+
+Benchmark result:
+{
+    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
+    "context": {
+        "benchmark_language": "Python"
+    },
+    "github": {
+        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
+        "repository": "https://github.com/conbench/conbench"
+    },
+    "info": {
+        "benchmark_language_version": "Python 3.9.7"
+    },
+    "machine_info": {
+        "architecture_name": "arm64",
+        "cpu_core_count": "8",
+        "cpu_frequency_max_hz": "0",
+        "cpu_l1d_cache_bytes": "65536",
+        "cpu_l1i_cache_bytes": "131072",
+        "cpu_l2_cache_bytes": "4194304",
+        "cpu_l3_cache_bytes": "0",
+        "cpu_model_name": "Apple M1",
+        "cpu_thread_count": "8",
+        "gpu_count": "0",
+        "gpu_product_names": [],
+        "kernel_name": "20.6.0",
+        "memory_bytes": "17179869184",
+        "name": "diana",
+        "os_name": "macOS",
+        "os_version": "11.5.2"
+    },
+    "stats": {
+        "data": [
+            "0.000004"
+        ],
+        "iqr": "0.000000",
+        "iterations": 1,
+        "max": "0.000004",
+        "mean": "0.000004",
+        "median": "0.000004",
+        "min": "0.000004",
+        "q1": "0.000004",
+        "q3": "0.000004",
+        "stdev": 0,
+        "time_unit": "s",
+        "times": [],
+        "unit": "s"
+    },
+    "tags": {
+        "columns": "10",
+        "name": "matrix",
+        "rows": "2"
+    }
+}
+
+Benchmark result:
+{
+    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
+    "run_id": "48acd853b8294df9a1f5457f192456f3",
+    "timestamp": "2021-11-09T22:15:23.509211+00:00",
+    "context": {
+        "benchmark_language": "Python"
+    },
+    "github": {
+        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
+        "repository": "https://github.com/conbench/conbench"
+    },
+    "info": {
+        "benchmark_language_version": "Python 3.9.7"
+    },
+    "machine_info": {
+        "architecture_name": "arm64",
+        "cpu_core_count": "8",
+        "cpu_frequency_max_hz": "0",
+        "cpu_l1d_cache_bytes": "65536",
+        "cpu_l1i_cache_bytes": "131072",
+        "cpu_l2_cache_bytes": "4194304",
+        "cpu_l3_cache_bytes": "0",
+        "cpu_model_name": "Apple M1",
+        "cpu_thread_count": "8",
+        "gpu_count": "0",
+        "gpu_product_names": [],
+        "kernel_name": "20.6.0",
+        "memory_bytes": "17179869184",
+        "name": "diana",
+        "os_name": "macOS",
+        "os_version": "11.5.2"
+    },
+    "stats": {
+        "data": [
+            "0.000002"
+        ],
+        "iqr": "0.000000",
+        "iterations": 1,
+        "max": "0.000002",
+        "mean": "0.000002",
+        "median": "0.000002",
+        "min": "0.000002",
+        "q1": "0.000002",
+        "q3": "0.000002",
+        "stdev": 0,
+        "time_unit": "s",
+        "times": [],
+        "unit": "s"
+    },
+    "tags": {
+        "columns": "2",
+        "name": "matrix",
+        "rows": "10"
+    }
+}
+```
+
+### Example R benchmarks
+
+Here are a few examples illustrating how to integrate R benchmarks with
+Conbench.
+
+The first one just times `1 + 1` in R, and the second one executes an R
+benchmark from a library of R benchmarks (in this case
+[arrowbench](https://github.com/ursacomputing/arrowbench)).
+
+If you find yourself wrapping a lot of R benchmarks in Python to integrate them
+with Conbench (to get uniform JSON benchmark results which you can persist and
+publish on a Conbench server), you'll probably want to extract much of the
+boilerplate out into a base class.
+
+
+```python
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmarkR(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that records an R benchmark result."""
+
+    external = True
+    name = "external-r"
+
+    def run(self, **kwargs):
+        result, output = self._run_r_command()
+        info, context = self.conbench.get_r_info_and_context()
+
+        yield self.conbench.record(
+            {"data": [result], "unit": "s"},
+            self.name,
+            info=info,
+            context=context,
+            options=kwargs,
+            output=output,
+        )
+
+    def _run_r_command(self):
+        output, _ = self.conbench.execute_r_command(self._get_r_command())
+        result = float(output.split("\n")[-1].split("[1] ")[1])
+        return result, output
+
+    def _get_r_command(self):
+        return (
+            f"addition <- function() { 1 + 1 }; "
+            "start_time <- Sys.time();"
+            "addition(); "
+            "end_time <- Sys.time(); "
+            "result <- end_time - start_time; "
+            "as.numeric(result); "
+        )
+```
+
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench external-r --help
+
+Usage: conbench external-r [OPTIONS]
+
+  Run external-r benchmark.
+
+Options:
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+```
+
+
+```python
+import json
+
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmarkOptionsR(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that records an R benchmark result (with options)."""
+
+    external = True
+    name = "external-r-options"
+    options = {
+        "iterations": {"default": 1, "type": int},
+        "drop_caches": {"type": bool, "default": "false"},
+    }
+
+    def run(self, **kwargs):
+        data, iterations = [], kwargs.get("iterations", 1)
+        info, context = self.conbench.get_r_info_and_context()
+
+        for _ in range(iterations):
+            if kwargs.get("drop_caches", False):
+                self.conbench.sync_and_drop_caches()
+            result, output = self._run_r_command()
+            data.append(result["result"][0]["real"])
+
+        yield self.conbench.record(
+            {"data": data, "unit": "s"},
+            self.name,
+            info=info,
+            context=context,
+            options=kwargs,
+            output=output,
+        )
+
+    def _run_r_command(self):
+        r_command = self._get_r_command()
+        self.conbench.execute_r_command(r_command)
+        with open("placebo.json") as json_file:
+            data = json.load(json_file)
+        return data, json.dumps(data, indent=2)
+
+    def _get_r_command(self):
+        return (
+            "library(arrowbench); "
+            "out <- run_one(arrowbench:::placebo); "
+            "cat(jsonlite::toJSON(out), file='placebo.json'); "
+        )
+```
+
+```
+(conbench) $ cd ~/workspace/conbench/legacy/tests/benchmark/
+(conbench) $ conbench external-r --help
+
+Usage: conbench external-r-options [OPTIONS]
+
+  Run external-r-options benchmark.
+
+Options:
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+```
diff --git a/legacy/conbenchlegacy/__init__.py b/legacy/conbenchlegacy/__init__.py
new file mode 100644
index 000000000..26d23badb
--- /dev/null
+++ b/legacy/conbenchlegacy/__init__.py
@@ -0,0 +1,3 @@
+from ._version import __version__
+
+__all__ = ["__version__"]
diff --git a/legacy/conbenchlegacy/_version.py b/legacy/conbenchlegacy/_version.py
new file mode 100644
index 000000000..18449c13e
--- /dev/null
+++ b/legacy/conbenchlegacy/_version.py
@@ -0,0 +1,1 @@
+__version__ = "2023.4.10"
diff --git a/legacy/conbenchlegacy/cli.py b/legacy/conbenchlegacy/cli.py
new file mode 100755
index 000000000..8c7da07de
--- /dev/null
+++ b/legacy/conbenchlegacy/cli.py
@@ -0,0 +1,171 @@
+import json
+
+import click
+
+from . import __version__
+from .runner import LIST, REGISTRY
+from .util import register_benchmarks
+
+register_benchmarks()
+BENCHMARKS = {}
+for benchmark in REGISTRY:
+    BENCHMARKS[benchmark.name] = benchmark
+
+
+if not BENCHMARKS:
+    click.echo(click.style("\nNo benchmarks registered.", fg="red"))
+    tip_msg = "(run conbench from a directory containing benchmarks).\n"
+    click.echo(click.style(tip_msg, fg="yellow"))
+
+
+@click.group()
+def conbench():
+    """Conbench: Language-independent Continuous Benchmarking (CB) Framework"""
+    pass
+
+
+@conbench.command(name="list")
+def list_benchmarks():
+    """List of benchmarks (for orchestration)."""
+    benchmarks = []
+    if LIST:
+        benchmarks = LIST[0]().list(BENCHMARKS)
+    print(json.dumps(benchmarks, indent=2))
+
+
+@conbench.command(name="version")
+def version():
+    """Display Conbench version."""
+    print(f"conbench version: {__version__}")
+
+
+def _option(params, name, default, _type, help_msg=None):
+    params.append(
+        click.Option(
+            (name,),
+            type=_type,
+            default=default,
+            help=help_msg,
+            show_default=True,
+        )
+    )
+
+
+def _choice(params, name, choices):
+    params.append(
+        click.Option(
+            (name,),
+            default=str(choices[0]),
+            show_default=False,
+            type=click.Choice([str(c) for c in choices], case_sensitive=False),
+        )
+    )
+
+
+def _help(benchmark, name, cases, tags):
+    if not cases:
+        description = getattr(benchmark, "description", f"Run {name} benchmark.")
+        return description
+
+    valid = [list(zip(tags, case)) for case in cases]
+    benchmarks = [[f"{option}={value}" for option, value in case] for case in valid]
+    examples = "\b\n".join([" ".join(example) for example in benchmarks])
+    description = getattr(benchmark, "description", f"Run {name} benchmark(s).")
+    examples = f"\b\nValid benchmark combinations:\n{examples}"
+    note = "For each benchmark option, the first option value is the default."
+    all_cases = f"\b\nTo run all combinations:\n$ conbench {name} --all=true"
+    return f"{description}\n\n{note}\n\n{examples}\n\n{all_cases}"
+
+
+def _to_cli_name(name):
+    return f"--{name.replace('_', '-')}"
+
+
+for name, benchmark in BENCHMARKS.items():
+    params = []
+
+    instance = benchmark()
+    fields, cases = instance.fields, instance.cases
+    options = getattr(benchmark, "options", {})
+    arguments = getattr(benchmark, "arguments", [])
+    tags = [_to_cli_name(tag) for tag in fields]
+    external = getattr(benchmark, "external", False)
+
+    for k, v in instance.case_options.items():
+        _choice(params, _to_cli_name(k), sorted(v))
+
+    for argument in arguments:
+        params.append(click.Argument((argument,)))
+    if cases:
+        _option(params, "--all", "false", bool)
+    for option, config in options.items():
+        if "choices" in config:
+            _choice(params, _to_cli_name(option), config["choices"])
+        else:
+            help_msg = config.get("help", None)
+            _option(
+                params,
+                _to_cli_name(option),
+                config.get("default"),
+                config["type"],
+                help_msg=help_msg,
+            )
+    if not external:
+        _option(params, "--iterations", 1, int)
+        _option(params, "--drop-caches", "false", bool)
+        _option(params, "--gc-collect", "true", bool)
+        _option(params, "--gc-disable", "true", bool)
+    _option(params, "--show-result", "true", bool)
+    _option(params, "--show-output", "false", bool)
+
+    params.append(
+        click.Option(
+            ("--run-id",),
+            type=str,
+            help="Group executions together with a run id.",
+        )
+    )
+    params.append(
+        click.Option(
+            ("--run-name",),
+            type=str,
+            help="Free-text name of run (commit ABC, pull request 123, etc).",
+        )
+    )
+    params.append(
+        click.Option(
+            ("--run-reason",),
+            type=str,
+            help="Low-cardinality reason for run (commit, pull request, manual, etc).",
+        )
+    )
+
+    def _benchmark(
+        show_result,
+        show_output,
+        benchmark=benchmark,
+        **kwargs,
+    ):
+        for result, output in benchmark().run(**kwargs):
+            if show_output:
+                click.echo()
+                click.echo(click.style("Benchmark output:", fg="yellow"))
+                click.echo(click.style(str(output), fg="blue"))
+            if show_result:
+                click.echo()
+                click.echo(click.style("Benchmark result:", fg="yellow"))
+                json_result = json.dumps(result, indent=4, sort_keys=True)
+                if "error" in json_result:
+                    click.echo(click.style(json_result, fg="red"))
+                else:
+                    click.echo(click.style(json_result, fg="green"))
+            del output
+
+    conbench.add_command(
+        click.Command(
+            name,
+            params=params,
+            callback=_benchmark,
+            help=_help(benchmark, name, cases, tags),
+        )
+    )
diff --git a/legacy/conbenchlegacy/machine_info.py b/legacy/conbenchlegacy/machine_info.py
new file mode 100644
index 000000000..4c30984af
--- /dev/null
+++ b/legacy/conbenchlegacy/machine_info.py
@@ -0,0 +1,357 @@
+import os
+import platform
+import re
+import subprocess
+import warnings
+
+
+def _sysctl(stat):
+    return ["sysctl", "-n", stat]
+
+
+MUST_BE_INTS = [
+    "cpu_core_count",
+    "cpu_thread_count",
+    "cpu_frequency_max_hz",
+    "cpu_l1d_cache_bytes",
+    "cpu_l1i_cache_bytes",
+    "cpu_l2_cache_bytes",
+    "cpu_l3_cache_bytes",
+    "memory_bytes",
+    "gpu_count",
+]
+
+
+COMMANDS = {
+    "kernel_name": ["uname", "-r"],
+    "cpu_model_name": _sysctl("machdep.cpu.brand_string"),
+    "cpu_core_count": _sysctl("hw.physicalcpu"),
+    "cpu_thread_count": _sysctl("hw.logicalcpu"),
+    "cpu_frequency_max_hz": _sysctl("hw.cpufrequency_max"),
+    "cpu_l1d_cache_bytes": _sysctl("hw.l1dcachesize"),
+    "cpu_l1i_cache_bytes": _sysctl("hw.l1icachesize"),
+    "cpu_l2_cache_bytes": _sysctl("hw.l2cachesize"),
+    "cpu_l3_cache_bytes": _sysctl("hw.l3cachesize"),
+    "memory_bytes": _sysctl("hw.memsize"),
+}
+
+
+LSCPU_MAPPING = {
+    "cpu_model_name": "Model name",
+    "cpu_frequency_max_hz": "CPU max MHz",
+    "cpu_l1d_cache_bytes": "L1d cache",
+    "cpu_l1i_cache_bytes": "L1i cache",
+    "cpu_l2_cache_bytes": "L2 cache",
+    "cpu_l3_cache_bytes": "L3 cache",
+}
+
+MEMINFO_MAPPING = {
+    "memory_bytes": "MemTotal",
+}
+
+NVIDIA_SMI_MAPPING = {
+    "gpu_count": None,
+    "gpu_product_names": None,
+}
+
+CPUINFO_MAPPING = {
+    "cpu_model_name": "brand_raw",
+    "cpu_l1d_cache_bytes": "l1_data_cache_size",
+    "cpu_l1i_cache_bytes": "l1_instruction_cache_size",
+    "cpu_l2_cache_bytes": "l2_cache_size",
+    "cpu_l3_cache_bytes": "l3_cache_size",
+}
+
+
+def python_info():
+    version = _exec_command(["python", "--version"])
+    return {
+        "benchmark_language": "Python",
+        "benchmark_language_version": version,
+    }
+
+
+def r_info():
+    r = "cat(version[['version.string']], '\n')"
+    version = _exec_command(["R", "-s", "-q", "-e", r])
+    return {
+        "benchmark_language": "R",
+        "benchmark_language_version": version,
+    }
+
+
+class GitParseWarning(RuntimeWarning):
+    pass
+
+
+def github_info():
+    """Attempts to inspect a locally cloned repository for git information that can be
+    posted to the "github" key when creating a run.
+
+    It's recommended to NOT use this function, and instead manually supply the commit
+    and repository (in the form "org/repo"). When doing so, ignore the "branch" key
+    and only supply the "pr_number" (or leave it None if it's a commit to the default
+    branch).
+    """
+    commit = _exec_command(["git", "rev-parse", "HEAD"])
+    if not commit:
+        warnings.warn(
+            "error in github_info(): probably not in a git repo; returning {}",
+            GitParseWarning,
+        )
+        return {}
+
+    branches = _exec_command(["git", "branch", "-vv"])
+    if "* (HEAD detached" in branches:
+        remote = "origin"
+        branch = "<DETATCHED BRANCH>"
+        warnings.warn(
+            f"error in github_info(): detached HEAD; returning {branch=}",
+            GitParseWarning,
+        )
+
+    else:
+        current_branch = [b for b in branches.split("\n") if b.startswith("*")][0]
+        _, branch_name, _, upstream, *_ = current_branch.split()
+        if not upstream.startswith("["):
+            remote = "origin"
+            branch = _exec_command(["git", "branch", "--show-current"])
+            warnings.warn(
+                f"error in github_info(): untracked branch; returning {branch=}",
+                GitParseWarning,
+            )
+        else:
+            remote = upstream[1:].split("/")[0]
+            branch = branch_name
+
+    remote_url = _exec_command(["git", "remote", "get-url", remote])
+    fork = re.search("(?<=.com[/:])([^/]*?)(?=/)", remote_url).group(0)
+
+    return {
+        "commit": commit,
+        "repository": remote_url.rsplit(".git")[0],
+        "branch": f"{fork}:{branch}",
+        "pr_number": None,
+    }
+
+
+def machine_info(host_name):
+    os_name, os_version = platform.platform(terse=True).split("-", maxsplit=1)
+
+    host_name = host_name or os.environ.get("CONBENCH_MACHINE_INFO_NAME")
+    host_name = host_name or platform.node()
+
+    info = {
+        "name": host_name,
+        "os_name": os_name,
+        "os_version": os_version,
+        "architecture_name": platform.machine(),
+        "kernel_name": None,
+        "memory_bytes": None,
+        "cpu_model_name": None,
+        "cpu_core_count": None,
+        "cpu_thread_count": None,
+        "cpu_l1d_cache_bytes": None,
+        "cpu_l1i_cache_bytes": None,
+        "cpu_l2_cache_bytes": None,
+        "cpu_l3_cache_bytes": None,
+        "cpu_frequency_max_hz": None,
+        "gpu_count": None,
+        "gpu_product_names": [],
+    }
+
+    _commands(info)
+    _meminfo(info)
+    _lscpu(info)
+    _cpuinfo(info)
+    _psutil(info)
+    _nvidia_smi(info)
+
+    for key in MUST_BE_INTS:
+        try:
+            int(info[key])
+        except (ValueError, TypeError):
+            info[key] = 0
+
+    info["memory_bytes"] = _round_memory(int(info["memory_bytes"]))
+
+    for key in info:
+        if not isinstance(info[key], list):
+            info[key] = str(info[key])
+
+    return info
+
+
+def _round_memory(value):
+    # B -> GiB -> B
+    gigs = 1024**3
+    return int("{:.0f}".format(value / gigs)) * gigs
+
+
+def _commands(info):
+    for key, command in COMMANDS.items():
+        try:
+            result = _exec_command(command)
+            info[key] = result if result else ""
+        except:
+            info[key] = ""
+
+
+def _psutil(info):
+    # TODO: rename to cpu_count_logical
+    if not info["cpu_thread_count"]:
+        info["cpu_thread_count"] = os.cpu_count()
+
+    try:
+        import psutil
+
+        # TODO: rename to cpu_count_physical
+        if not info["cpu_core_count"]:
+            info["cpu_core_count"] = psutil.cpu_count(logical=False)
+    except:
+        pass
+
+
+def _cpuinfo(info):
+    missing = _has_missing(info, CPUINFO_MAPPING)
+    if not missing:
+        return
+
+    try:
+        import cpuinfo
+
+        cpu_info = cpuinfo.get_cpu_info()
+        _fill_from_cpuinfo(info, cpu_info)
+    except:
+        pass
+
+
+def _lscpu(info):
+    missing = _has_missing(info, LSCPU_MAPPING)
+    if not missing:
+        return
+
+    try:
+        command = ["lscpu", "--bytes"]
+        result = subprocess.run(command, capture_output=True)
+        if result.returncode != 0:
+            return
+    except:
+        return
+
+    parts = result.stdout.decode("utf-8").strip().split("\n")
+    _fill_from_lscpu(info, parts)
+
+
+def _meminfo(info):
+    missing = _has_missing(info, MEMINFO_MAPPING)
+    if not missing:
+        return
+
+    try:
+        command = ["cat", "/proc/meminfo"]
+        result = subprocess.run(command, capture_output=True)
+        if result.returncode != 0:
+            return
+    except:
+        return
+
+    parts = result.stdout.decode("utf-8").strip().split("\n")
+    _fill_from_meminfo(info, parts)
+
+
+def _nvidia_smi(info):
+    missing = _has_missing(info, NVIDIA_SMI_MAPPING)
+    if not missing:
+        return
+
+    try:
+        command = ["nvidia-smi", "--query-gpu=gpu_name", "--format=csv,noheader"]
+        result = subprocess.run(command, capture_output=True)
+        if result.returncode != 0:
+            return
+    except:
+        return
+
+    parts = result.stdout.decode("utf-8").strip().split("\n")
+    if parts:
+        info["gpu_count"] = len(parts)
+        info["gpu_product_names"] = parts
+
+
+def _fill_from_cpuinfo(info, cpu_info):
+    for key, lookup in CPUINFO_MAPPING.items():
+        if lookup not in cpu_info:
+            continue
+        try:
+            if not info[key]:
+                if key == "cpu_model_name":
+                    info[key] = cpu_info[lookup]
+                else:
+                    value = str(cpu_info[lookup])
+                    parts = value.split(" ")
+                    if len(parts) == 1:
+                        info[key] = int(float(value))
+                    elif len(parts) == 2:
+                        value, unit = parts
+                        value = int(float(value))
+                        if unit == "KiB":
+                            info[key] = value * 1024
+                        elif unit == "MiB":
+                            info[key] = value * 1024 * 1024
+        except ValueError:
+            pass
+
+
+def _fill_from_lscpu(info, parts):
+    lscpu_dict = {}
+    for part in parts:
+        x = part.split(":")
+        if len(x) == 2:
+            k, v = x
+            lscpu_dict[k.strip()] = v.strip()
+
+    for key, lookup in LSCPU_MAPPING.items():
+        if lookup not in lscpu_dict:
+            continue
+        try:
+            if not info[key]:
+                if lookup.endswith("MHz"):
+                    info[key] = int(float(lscpu_dict[lookup]) * 10**6)
+                elif key in MUST_BE_INTS:
+                    # lscpu sometimes includes descriptions after the value so remove that
+                    info[key] = int(float(lscpu_dict[lookup].split(maxsplit=1)[0]))
+                else:
+                    info[key] = lscpu_dict[lookup]
+        except ValueError:
+            pass
+
+
+def _fill_from_meminfo(info, parts):
+    meminfo_dict = {}
+    for part in parts:
+        x = part.split(":")
+        if len(x) == 2:
+            k, v = x
+            meminfo_dict[k.strip()] = v.strip().split(" ")[0]
+
+    for key, lookup in MEMINFO_MAPPING.items():
+        if lookup not in meminfo_dict:
+            continue
+        try:
+            if not info[key]:
+                info[key] = int(float(meminfo_dict[lookup]) * 1000)
+        except ValueError:
+            pass
+
+
+def _has_missing(info, mapping):
+    for key in mapping:
+        if not info[key]:
+            return True
+    return False
+
+
+def _exec_command(command):
+    result = subprocess.run(command, capture_output=True)
+    return result.stdout.decode("utf-8").strip()
diff --git a/legacy/conbenchlegacy/runner.py b/legacy/conbenchlegacy/runner.py
new file mode 100644
index 000000000..ec3f9c8bc
--- /dev/null
+++ b/legacy/conbenchlegacy/runner.py
@@ -0,0 +1,467 @@
+import abc
+import collections
+import datetime
+import functools
+import gc
+import statistics
+import subprocess
+import time
+import traceback
+import uuid
+from typing import Any, Optional, Tuple
+
+import numpy as np
+
+from .machine_info import github_info, machine_info, python_info, r_info
+from .util import Connection
+
+REGISTRY = []
+LIST = []  # type: ignore[var-annotated]
+
+
+LANG = "benchmark_language"
+LANG_VERSION = "benchmark_language_version"
+
+
+def register_benchmark(cls):
+    REGISTRY.append(cls)
+    return cls
+
+
+def register_list(cls):
+    if not LIST:
+        LIST.append(cls)
+    return cls
+
+
+def _now_formatted():
+    now = datetime.datetime.now(datetime.timezone.utc)
+    return now.isoformat()
+
+
+class Benchmark(abc.ABC):
+    def __init__(self):
+        self.conbench = Conbench()
+
+    @abc.abstractmethod
+    def run(self, **kwargs):
+        """
+        Parameters
+        ----------
+        all : boolean, optional
+            Run all benchmark cases.
+        case : sequence, optional
+            Benchmark options sequence (rather than individual params):
+            [<option1>, <option2>, ..., <option3>]
+        iterations : int, default 1
+            Number of times to run the benchmark.
+        drop_caches : boolean, default False
+            Whether to drop caches before each benchmark run.
+        gc_collect : boolean, default True
+            Whether to do garbage collection before each benchmark run.
+        gc_disable : boolean, default True
+            Whether to do disable collection during each benchmark run.
+        run_id : str, optional
+            Group executions together with a run id.
+        run_name : str, optional
+            Free-text name of run (commit ABC, pull request 123, etc).
+        run_reason : str, optional
+            Reason for run (commit, pull request, manual, etc). Probably will be used
+            to group runs, so try to keep the cardinality low.
+        Returns
+        -------
+        (result, output) : sequence
+            result : The benchmark result.
+            output : The output from the benchmarked function.
+        """
+        pass
+
+    @property
+    def cases(self):
+        cases = getattr(self, "valid_cases", [])
+        return cases[1:] if cases else []
+
+    @property
+    def fields(self):
+        cases = getattr(self, "valid_cases", [])
+        return cases[0] if cases else []
+
+    @property
+    def case_options(self):
+        options = collections.defaultdict(set)
+        fields, cases = self.fields, self.cases
+        if not cases:
+            return options
+        for case in cases:
+            for k, v in zip(fields, case):
+                options[k].add(v)
+        return options
+
+    @property
+    def case_ids(self):
+        return [", ".join([str(c) for c in case]) for case in self.cases]
+
+    def get_cases(self, case, options):
+        run_all = options.get("all", False)
+        return self.cases if run_all else [self._get_case(case, options)]
+
+    def _get_case(self, case, options):
+        if case is None:
+            case = []
+            for i, c in enumerate(self.fields):
+                pick = self.cases[0][i]
+                if isinstance(pick, bool):
+                    case.append(options.get(c).lower() in ["True", "true"])
+                else:
+                    t = type(self.cases[0][i])
+                    case.append(t(options.get(c)))
+        case_tuples = [tuple(c) for c in self.cases]
+        if tuple(case) not in case_tuples:
+            invalid_case = dict(zip(self.fields, case))
+            raise ValueError(f"Invalid case: {invalid_case}")
+        return case
+
+
+class BenchmarkList(abc.ABC):
+    @abc.abstractmethod
+    def list(self, classes):
+        pass
+
+
+class MixinPython:
+    @functools.cached_property
+    def python_info(self):
+        return python_info()
+
+    def set_python_info_and_context(self, info, context):
+        lang = self.python_info
+        info.update({LANG_VERSION: lang[LANG_VERSION]})
+        context.update({LANG: lang[LANG]})
+
+    def benchmark(self, f, name, publish=True, **kwargs):
+        """Benchmark a function and publish the result."""
+        (
+            tags,
+            optional_benchmark_info,
+            context,
+            info,
+            github,
+            options,
+            cluster_info,
+            _,
+        ) = self._init(kwargs)
+        self.set_python_info_and_context(info, context)
+
+        timing_options = self._get_timing_options(options)
+        iterations = timing_options.pop("iterations")
+        if iterations < 1:
+            raise ValueError(f"Invalid iterations: {iterations}")
+
+        try:
+            data, output = self._get_timing(f, iterations, timing_options)
+            # It's hard to read what this next function call really does. It
+            # does _not_ publish, but I think it creates a specific data
+            # structure. Should this be in the exception handler? Within
+            # self._get_timing() above we run user-given code, so that is
+            # expected to raise exceptions, and wants to be handled. But which
+            # exceptions is  self.record() expected to raise especially when
+            # _not_ doing HTTP interaction? And why do we handle those
+            # exceptions in the same way as those exceptions that are raised by
+            # user-given code?
+            benchmark, _ = self.record(
+                {"data": data, "unit": "s"},
+                name,
+                tags=tags,
+                optional_benchmark_info=optional_benchmark_info,
+                context=context,
+                info=info,
+                github=github,
+                options=options,
+                cluster_info=cluster_info,
+                publish=False,
+            )
+        except Exception as exc:
+            error = {"stack_trace": traceback.format_exc()}
+            benchmark, _ = self.record(
+                None,
+                name,
+                tags=tags,
+                optional_benchmark_info=optional_benchmark_info,
+                context=context,
+                info=info,
+                github=github,
+                options=options,
+                cluster_info=cluster_info,
+                error=error,
+                publish=False,
+            )
+            raise exc
+        finally:
+            if publish:
+                # It's a bit unclear -- is `benchmark` defined in _all_ cases
+                # when we arrive here?
+                # https://pylint.readthedocs.io/en/latest/user_guide/messages/error/used-before-assignment.html
+                self.publish(benchmark)  # pylint: disable=used-before-assignment
+        return benchmark, output
+
+
+class MixinR:
+    @functools.cached_property
+    def r_info(self):
+        return r_info()
+
+    def get_r_info_and_context(self):
+        lang = self.r_info
+        info = {LANG_VERSION: lang[LANG_VERSION]}
+        context = {LANG: lang[LANG]}
+        return info, context
+
+    def execute_r_command(self, r_command, quiet=True):
+        if quiet:
+            command = ["R", "-s", "-q", "-e", r_command]
+        else:
+            command = ["R", "-e", r_command]
+        result = subprocess.run(command, capture_output=True)
+        output = result.stdout.decode("utf-8").strip()
+        error = result.stderr.decode("utf-8").strip()
+        if result.returncode != 0:
+            raise Exception(error)
+        return output, error
+
+
+class Conbench(Connection, MixinPython, MixinR):
+    def __init__(self):
+        super().__init__()
+        self._run_id = uuid.uuid4().hex
+        self._batch_id = uuid.uuid4().hex
+        self._drop_caches_failed = False
+        self._purge_failed = False
+
+    @functools.cached_property
+    def github_info(self):
+        return github_info()
+
+    @functools.cached_property
+    def machine_info(self):
+        return machine_info(self.config.host_name)
+
+    def record(
+        self, result, name, error=None, publish=True, **kwargs
+    ) -> Tuple[dict, Optional[Any]]:
+        """
+        Build up a dictionary in compliance with Conbench's benchmark result
+        JSON schema.
+
+        Publish that benchmark result to the Conbench HTTP API (if `publish` is
+        True).
+
+        Return the dictionary.
+
+        The `name` parameter is here for legacy reasons and allows for passing
+        the name of the conceptual benchmark; "the benchmark name".
+
+        The benchmark name can however as of today also be provided via
+        `tags['name']` in `kwargs`. Throw ValueError upon conflict when both
+        are provided.
+        """
+        (
+            tags,
+            optional_benchmark_info,
+            context,
+            info,
+            github,
+            options,
+            cluster_info,
+            output,
+        ) = self._init(kwargs)
+
+        if (tags.get("name") is not None) and (name != tags["name"]):
+            raise ValueError(
+                f"`name` and `tags[\"name\"] are both supplied and do not match: {name=} != {tags['name']=}"
+            )
+
+        # Make sure that `name` key is in tags; currently the canonical way to
+        # communicate the name of the benchmark that this result has been
+        # obtained for.
+        tags["name"] = name
+
+        batch_id = options.get("batch_id")
+        if not batch_id:
+            batch_id = self._batch_id
+
+        run_id = options.get("run_id")
+        if not run_id:
+            run_id = self._run_id
+
+        # Make this naming challenge explicit.
+        optional_result_info = optional_benchmark_info
+
+        benchmark_result = {
+            "run_id": run_id,
+            "batch_id": batch_id,
+            "timestamp": _now_formatted(),
+            "context": context,
+            "info": info,
+            "tags": tags,
+            "optional_benchmark_info": optional_result_info,
+            "github": github,
+        }
+        if error:
+            benchmark_result["error"] = error
+
+        if result and result["data"]:
+            benchmark_result["stats"] = self._stats(
+                result["data"],
+                result["unit"],
+                result.get("times", []),
+                result.get("time_unit", "s"),
+            )
+
+        if cluster_info:
+            benchmark_result["cluster_info"] = cluster_info
+        else:
+            benchmark_result["machine_info"] = self.machine_info
+
+        run_name = options.get("run_name")
+        if run_name is not None:
+            benchmark_result["run_name"] = run_name
+
+        run_reason = options.get("run_reason")
+        if run_reason is not None:
+            benchmark_result["run_reason"] = run_reason
+
+        if publish:
+            self.publish(benchmark_result)
+
+        return benchmark_result, output
+
+    def get_run_id(self, options):
+        run_id = options.get("run_id")
+        return run_id if run_id else self._run_id
+
+    def mark_new_batch(self):
+        self._batch_id = uuid.uuid4().hex
+
+    def manually_batch(self, batch_id):
+        self._batch_id = batch_id
+
+    def sync_and_drop_caches(self):
+        if not self._drop_caches_failed:
+            command = "sync; echo 3 | sudo tee /proc/sys/vm/drop_caches"
+            try:
+                subprocess.check_output(command, shell=True)
+                return True
+            except subprocess.CalledProcessError:
+                self._drop_caches_failed = True
+
+        if not self._purge_failed:
+            command = "sync; sudo purge"
+            try:
+                subprocess.check_output(command, shell=True)
+                return True
+            except subprocess.CalledProcessError:
+                self._purge_failed = True
+
+        return False
+
+    def _init(self, kwargs):
+        tags = kwargs.get("tags", {})
+        optional_benchmark_info = kwargs.get("optional_benchmark_info", {})
+        context = kwargs.get("context", {})
+        info = kwargs.get("info", {})
+        github = kwargs.get("github", {})
+        options = kwargs.get("options", {})
+        cluster_info = kwargs.get("cluster_info", {})
+        github = github if github else self.github_info
+        return (
+            tags,
+            optional_benchmark_info,
+            context,
+            info,
+            github,
+            options,
+            cluster_info,
+            kwargs.get("output"),
+        )
+
+    def _get_timing(self, f, iterations, options):
+        times, output = [], None
+
+        for _ in range(iterations):
+            if options["drop_caches"]:
+                self.sync_and_drop_caches()
+            if options["gc_collect"]:
+                gc.collect()
+            if options["gc_disable"]:
+                gc.disable()
+
+            if output is not None:
+                # We only need output from the final run.
+                # For other runs delete before we run the next
+                # iteration to avoid doubling memory
+                del output
+
+            iteration_start = time.monotonic()
+            output = f()
+            times.append(time.monotonic() - iteration_start)
+
+            gc.enable()
+
+        return times, output
+
+    def _get_timing_options(self, options):
+        return {
+            "gc_collect": options.get("gc_collect", True),
+            "gc_disable": options.get("gc_disable", True),
+            "drop_caches": options.get("drop_caches", False),
+            "iterations": options.get("iterations", 1),
+        }
+
+    @staticmethod
+    def _stats(data, unit, times, time_unit, one_sample_no_mean=False):
+        fmt = "{:.6f}"
+
+        def _format(f, data, min_length=0):
+            return fmt.format(f(data)) if len(data) > min_length else 0
+
+        if not data:
+            raise ValueError(f"Invalid data: {data}")
+
+        q1, q3 = np.percentile(data, [25, 75])
+
+        # Note(JP): sending e.g. iqr and q1 for iterations == 1 does not
+        # make sense. The aggregates should be built by the API implementation
+        # unless we have more data here than we send, in which case
+        # `iterations` should be larger than 1 (or larger than 2). Also see
+        # https://github.com/voltrondata-labs/arrow-benchmarks-ci/issues/129
+        # https://github.com/conbench/conbench/issues/1298
+        # https://github.com/conbench/conbench/issues/533
+        # https://github.com/conbench/conbench/pull/1127
+        result = {
+            "data": [fmt.format(x) for x in data],
+            "times": [fmt.format(x) for x in times],
+            "unit": unit,
+            "time_unit": time_unit,
+            "iterations": len(data),
+            "mean": _format(statistics.mean, data),
+            "median": _format(statistics.median, data),
+            "min": _format(min, data),
+            "max": _format(max, data),
+            "stdev": _format(statistics.stdev, data, min_length=2),
+            "q1": fmt.format(q1),
+            "q3": fmt.format(q3),
+            "iqr": fmt.format(q3 - q1),
+        }
+
+        if one_sample_no_mean:
+            # Submit a BenchmarkResult in a history that has  one data point
+            # and no aggregates, in particular no (user-given) mean value set.
+            agg_keys = ("q1", "q3", "mean", "median", "min", "max", "stdev", "iqr")
+
+            result["data"] = [result["data"][0]]
+            result["times"] = [result["times"][0]] if result["times"] else []
+            result["iterations"] = 1
+            for k in agg_keys:
+                del result[k]
+
+        return result
diff --git a/legacy/conbenchlegacy/util.py b/legacy/conbenchlegacy/util.py
new file mode 100644
index 000000000..1eb5c46b3
--- /dev/null
+++ b/legacy/conbenchlegacy/util.py
@@ -0,0 +1,154 @@
+import json
+import logging
+import os
+import time
+import urllib.parse
+
+import click
+import requests
+import yaml
+from _pytest.pathlib import import_path
+from requests.adapters import HTTPAdapter
+
+try:
+    from urllib3.util import Retry
+except ImportError:
+    # Legacy around times where requests had version 2.3 something.
+    from requests.packages.urllib3.util.retry import Retry  # type: ignore[no-redef]
+
+retry_strategy = Retry(
+    total=5,
+    status_forcelist=[502, 503, 504],
+    allowed_methods=frozenset(["GET", "POST"]),
+    backoff_factor=4,  # will retry in 2, 4, 8, 16, 32 seconds
+)
+adapter = HTTPAdapter(max_retries=retry_strategy)
+
+
+log = logging.getLogger()
+
+
+class Connection:
+    def __init__(self):
+        self.config = Config(get_config())
+        self.session = None
+
+    def publish(self, benchmark):
+        self.post(self.config.benchmarks_url, benchmark)
+
+    def post(self, url, data):
+        if self.session:
+            # already authenticated, just do post
+            self._post(url, data, 201)
+
+        if not self.session:
+            # not already authenticated, or authentication expired (try again)
+            self._post(self.config.login_url, self.config.credentials, 204)
+            self._post(url, data, 201)
+
+    def _post(self, url, data, expected):
+        try:
+            if not self.session:
+                self.session = requests.Session()
+                self.session.mount("https://", adapter)
+            start = time.monotonic()
+            response = self.session.post(url, json=data)
+            log.info("Time to POST %s: %.5f s", url, time.monotonic() - start)
+            if response.status_code != expected:
+                self._unexpected_response("POST", response, url)
+        except requests.exceptions.ConnectionError:
+            self.session = None
+
+    def _unexpected_response(self, method, response, url):
+        self._print_error(f"\n{method} {url} failed", red=True)
+        if response.content:
+            try:
+                message = json.loads(response.content)
+                self._print_error(f"{json.dumps(message, indent=2)}\n")
+            except json.JSONDecodeError:
+                self._print_error(f"{response.content}\n")
+        self.session = None
+
+    def _print_error(self, msg, red=False):
+        if red:
+            click.echo(click.style(msg, fg="red"))
+        else:
+            click.echo(msg)
+
+
+def places_to_look():
+    current_dir = os.getcwd()
+    benchmarks_dir = os.path.join(current_dir, "benchmarks")
+    if os.path.exists(benchmarks_dir):
+        return [current_dir, benchmarks_dir]
+    return [current_dir]
+
+
+class Config:
+    def __init__(self, config):
+        url = config.get("url", "http://localhost:5000")
+        email = config.get("email", "conbench@example.com")
+        password = str(config.get("password", "conbench"))
+        self.host_name = config.get("host_name")
+        self.login_url = urllib.parse.urljoin(url, "api/login/")
+        self.benchmarks_url = urllib.parse.urljoin(url, "api/benchmarks/")
+        self.credentials = {"email": email, "password": password}
+
+
+def get_config(filename=None):
+    """Get config from a yaml file named .conbench in the current
+    working directory, or a sub directory called "benchmarks".
+    """
+    config = {}
+    if filename is None:
+        filename = ".conbench"
+    for directory in places_to_look():
+        file_path = os.path.join(directory, filename)
+        if os.path.exists(file_path):
+            with open(file_path) as f:
+                return yaml.load(f, Loader=yaml.FullLoader)
+    return config
+
+
+def register_benchmarks(directory=None):
+    """Look for files matching the following patterns in the current
+    working directory or a sub directory called "benchmarks", and
+    import them.
+
+        benchmark*.py
+        *benchmark.py
+        *benchmarks.py
+
+    This registers benchmarks that are decorated as conbench benchmarks.
+
+        import conbenchlegacy.runner
+
+        @conbenchlegacy.runner.register_benchmark
+        class ExampleBenchmark:
+            ...
+
+    It also registers the benchmark list class.
+
+        import conbenchlegacy.runner
+
+        @conbenchlegacy.runner.register_list
+        class ExampleBenchmarkList:
+            ...
+    """
+    dirs = places_to_look() if directory is None else [directory]
+    for directory in dirs:
+        with os.scandir(directory) as scan:
+            for entry in scan:
+                filename = entry.name
+                if (
+                    filename.startswith(".")
+                    or not entry.is_file()
+                    or not filename.endswith(".py")
+                ):
+                    continue
+                if (
+                    filename.startswith("benchmark")
+                    or filename.endswith("benchmark.py")
+                    or filename.endswith("benchmarks.py")
+                ):
+                    import_path(f"{directory}/{filename}", root=entry)
diff --git a/legacy/requirements-cli.txt b/legacy/requirements-cli.txt
new file mode 100644
index 000000000..86db30aa6
--- /dev/null
+++ b/legacy/requirements-cli.txt
@@ -0,0 +1,6 @@
+click
+numpy
+psutil
+py-cpuinfo
+PyYAML
+requests
diff --git a/legacy/requirements-dev.txt b/legacy/requirements-dev.txt
new file mode 100644
index 000000000..3ed85272d
--- /dev/null
+++ b/legacy/requirements-dev.txt
@@ -0,0 +1,3 @@
+conbench[server]
+Flask==2.2.3
+pytest>=7.0.0
diff --git a/legacy/setup.py b/legacy/setup.py
new file mode 100755
index 000000000..d1e65e8bf
--- /dev/null
+++ b/legacy/setup.py
@@ -0,0 +1,50 @@
+import pathlib
+
+import setuptools
+
+library_root = pathlib.Path(__file__).parent
+
+with open(library_root / "README.md", "r") as readme:
+    long_description = readme.read()
+
+__version__ = ""
+with open(library_root / "conbenchlegacy" / "_version.py", "r") as f:
+    exec(f.read())  # only overwrites the __version__ variable
+
+
+def parse_requirements(path: str):
+    return [
+        line.strip()
+        for line in pathlib.Path(__file__)
+        .parent.joinpath(path)
+        .read_text()
+        .splitlines()
+    ]
+
+
+requirements_cli = parse_requirements(path="requirements-cli.txt")
+requirements_dev = parse_requirements(path="requirements-dev.txt")
+
+setuptools.setup(
+    name="conbenchlegacy",
+    version=__version__,
+    description="Continuous Benchmarking (CB) Framework Legacy Library and CLI",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    packages=setuptools.find_packages(),
+    entry_points={
+        "console_scripts": [
+            "conbench=conbenchlegacy.cli:conbench",
+        ],
+    },
+    classifiers=[
+        "Programming Language :: Python :: 3.8",
+        "License :: OSI Approved :: MIT License",
+    ],
+    python_requires=">=3.8",
+    maintainer="Apache Arrow Developers",
+    maintainer_email="dev@arrow.apache.org",
+    url="https://github.com/conbench/conbench",
+    install_requires=requirements_cli,
+    extras_require={"dev": requirements_dev},
+)


Test patch:
diff --git a/legacy/tests/__init__.py b/legacy/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/legacy/tests/_fixtures.py b/legacy/tests/_fixtures.py
new file mode 100644
index 000000000..7f5c3572a
--- /dev/null
+++ b/legacy/tests/_fixtures.py
@@ -0,0 +1,206 @@
+from .helpers import _uuid
+
+CHILD = "02addad336ba19a654f9c857ede546331be7b631"
+PARENT = "4beb514d071c9beec69b8917b5265e77ade22fb3"
+GRANDPARENT = "6d703c4c7b15be630af48d5e9ef61628751674b2"
+ELDER = "81e9417eb68171e03a304097ae86e1fd83307130"
+
+REPO = "https://github.com/org/something"
+
+RESULTS_UP = [[1, 2, 3], [2, 3, 4], [10, 20, 30]]
+RESULTS_DOWN = [[10, 11, 12], [11, 12, 13], [1, 2, 3]]
+Z_SCORE_UP = 24.74873734152916  # Computed using statistics.stdev()
+Z_SCORE_DOWN = -13.435028842544401  # Computed using statistics.stdev()
+
+CLUSTER_INFO = {
+    "name": f"cluster-{_uuid()}",
+    "info": {"gpu": 1},
+    "optional_info": {"workers": 1},
+}
+
+MACHINE_INFO = {
+    "architecture_name": "x86_64",
+    "cpu_l1d_cache_bytes": "32768",
+    "cpu_l1i_cache_bytes": "32768",
+    "cpu_l2_cache_bytes": "262144",
+    "cpu_l3_cache_bytes": "4194304",
+    "cpu_core_count": "2",
+    "cpu_frequency_max_hz": "3500000000",
+    "cpu_model_name": "Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz",
+    "cpu_thread_count": "4",
+    "kernel_name": "19.6.0",
+    "memory_bytes": "17179869184",
+    "name": "diana",
+    "os_name": "macOS",
+    "os_version": "10.15.7",
+    "gpu_count": "2",
+    "gpu_product_names": ["Tesla T4", "GeForce GTX 1060 3GB"],
+}
+
+VALID_RESULT_PAYLOAD = {
+    "run_id": "2a5709d179f349cba69ed242be3e6321",
+    "run_name": "commit: 02addad336ba19a654f9c857ede546331be7b631",
+    "run_reason": "commit",
+    "batch_id": "7b2fdd9f929d47b9960152090d47f8e6",
+    "timestamp": "2020-11-25T21:02:44Z",
+    "context": {
+        "arrow_compiler_flags": "-fPIC -arch x86_64 -arch x86_64 -std=c++11 -Qunused-arguments -fcolor-diagnostics -O3 -DNDEBUG",
+        "benchmark_language": "Python",
+    },
+    "info": {
+        "arrow_version": "2.0.0",
+        "benchmark_language_version": "Python 3.8.5",
+        "arrow_compiler_version": "11.0.0.11000033",
+        "arrow_compiler_id": "AppleClang",
+    },
+    "optional_benchmark_info": {"trace_id": "some trace id", "logs": "some log uri"},
+    "validation": {"type": "pandas.testing", "success": True},
+    "github": {
+        "commit": "02addad336ba19a654f9c857ede546331be7b631",
+        "repository": "https://github.com/org/repo",
+        "branch": None,
+        "pr_number": 12345678,
+    },
+    "machine_info": MACHINE_INFO,
+    "stats": {
+        "data": [
+            "0.099094",
+            "0.037129",
+            "0.036381",
+            "0.148896",
+            "0.008104",
+            "0.005496",
+            "0.009871",
+            "0.006008",
+            "0.007978",
+            "0.004733",
+        ],
+        "times": [
+            "0.099094",
+            "0.037129",
+            "0.036381",
+            "0.148896",
+            "0.008104",
+            "0.005496",
+            "0.009871",
+            "0.006008",
+            "0.007978",
+            "0.004733",
+        ],
+        "unit": "s",
+        "time_unit": "s",
+        "iqr": "0.030442",
+        "iterations": 10,
+        "max": "0.1489",
+        "mean": "0.036369",
+        "median": "0.0089875",
+        "min": "0.004733",
+        "q1": "0.0065005",
+        "q3": "0.036942",
+        "stdev": "0.049194",
+    },
+    "tags": {
+        "compression": "snappy",
+        "cpu_count": "2",
+        "dataset": "nyctaxi_sample",
+        "file_type": "parquet",
+        "input_type": "arrow",
+        "name": "file-write",
+    },
+}
+
+VALID_RESULT_PAYLOAD_WITH_ERROR = dict(
+    run_id="ya5709d179f349cba69ed242be3e6323",
+    error={"stack_trace": "some trace", "command": "ls"},
+    **{
+        key: value
+        for key, value in VALID_RESULT_PAYLOAD.items()
+        if key not in ("stats", "run_id")
+    },
+)
+
+VALID_RESULT_PAYLOAD_WITH_ITERATION_ERROR = dict(
+    run_id="ya5709d179f349cba69ed242be3e6323",
+    # It's interesting that this has both, the `error` property set and
+    # missing iteration data. Either is sufficient for marking a result as
+    # "errored".
+    error={"stack_trace": "some trace", "command": "ls"},
+    **{
+        key: value
+        for key, value in VALID_RESULT_PAYLOAD.items()
+        if key not in ("stats", "run_id")
+    },
+    stats={
+        "data": [
+            "0.099094",
+            None,
+            "0.036381",
+            "0.148896",
+            "0.008104",
+            "0.005496",
+            "0.009871",
+            "0.006008",
+            "0.007978",
+            "0.004733",
+        ],
+        "times": [
+            "0.099094",
+            None,
+            "0.036381",
+            "0.148896",
+            "0.008104",
+            "0.005496",
+            "0.009871",
+            "0.006008",
+            "0.007978",
+            "0.004733",
+        ],
+        "unit": "s",
+        "time_unit": "s",
+        "iterations": 10,
+    },
+)
+
+VALID_RESULT_PAYLOAD_FOR_CLUSTER = dict(
+    run_id="3a5709d179f349cba69ed242be3e6323",
+    cluster_info=CLUSTER_INFO,
+    **{
+        key: value
+        for key, value in VALID_RESULT_PAYLOAD.items()
+        if key not in ("machine_info", "run_id")
+    },
+)
+
+VALID_RUN_PAYLOAD = {
+    "id": _uuid(),
+    "name": "commit: 02addad336ba19a654f9c857ede546331be7b631",
+    "reason": "commit",
+    "finished_timestamp": "2020-11-25T21:02:42Z",
+    "info": {
+        "setup": "passed",
+    },
+    "github": {
+        "commit": "02addad336ba19a654f9c857ede546331be7b631",
+        "repository": "https://github.com/org/repo",
+        "branch": None,
+        "pr_number": 12345678,
+    },
+    "machine_info": MACHINE_INFO,
+}
+
+VALID_RUN_PAYLOAD_FOR_CLUSTER = dict(
+    id=_uuid(),
+    cluster_info=CLUSTER_INFO,
+    **{
+        key: value
+        for key, value in VALID_RUN_PAYLOAD.items()
+        if key not in ("machine_info", "id")
+    },
+)
+
+VALID_RUN_PAYLOAD_WITH_ERROR = dict(
+    id=_uuid(),
+    error_info={"error": "error", "stack_trace": "stack_trace", "fatal": True},
+    error_type="fatal",
+    **{key: value for key, value in VALID_RUN_PAYLOAD.items() if key not in ("id")},
+)
diff --git a/legacy/tests/benchmark/__init__.py b/legacy/tests/benchmark/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/legacy/tests/benchmark/_example_benchmarks.py b/legacy/tests/benchmark/_example_benchmarks.py
new file mode 100644
index 000000000..b5a3b616f
--- /dev/null
+++ b/legacy/tests/benchmark/_example_benchmarks.py
@@ -0,0 +1,290 @@
+import json
+
+import conbenchlegacy.runner
+
+
+@conbenchlegacy.runner.register_list
+class BenchmarkList(conbenchlegacy.runner.BenchmarkList):
+    def list(self, classes):
+        benchmarks = []
+        for name, benchmark in classes.items():
+            instance, parts = benchmark(), [name]
+            if instance.cases:
+                parts.append("--all=true")
+            parts.append("--iterations=2")
+            benchmarks.append({"command": " ".join(parts)})
+        return sorted(benchmarks, key=lambda k: k["command"])
+
+
+@conbenchlegacy.runner.register_benchmark
+class SimpleBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark without cases."""
+
+    name = "addition"
+
+    def run(self, **kwargs):
+        yield self.conbench.benchmark(
+            self._get_benchmark_function(), self.name, options=kwargs
+        )
+
+    def _get_benchmark_function(self):
+        return lambda: 1 + 1
+
+
+@conbenchlegacy.runner.register_benchmark
+class SimpleBenchmarkWithClusterInfo(conbenchlegacy.runner.Benchmark):
+    """Example benchmark without cases."""
+
+    name = "product"
+
+    def run(self, **kwargs):
+        cluster_info = {
+            "name": "cluster 1",
+            "info": {"gpu": 1},
+            "optional_info": {"workers": 2},
+        }
+        yield self.conbench.benchmark(
+            self._get_benchmark_function(),
+            self.name,
+            cluster_info=cluster_info,
+            options=kwargs,
+        )
+
+    def _get_benchmark_function(self):
+        return lambda: 1 * 2
+
+
+@conbenchlegacy.runner.register_benchmark
+class CasesBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark with cases."""
+
+    name = "matrix"
+    valid_cases = (
+        ("rows", "columns"),
+        ("10", "10"),
+        ("2", "10"),
+        ("10", "2"),
+    )
+
+    def run(self, case=None, **kwargs):
+        for case in self.get_cases(case, kwargs):
+            rows, columns = case
+            tags = {"rows": rows, "columns": columns}
+            func = self._get_benchmark_function(rows, columns)
+            benchmark, output = self.conbench.benchmark(
+                func,
+                self.name,
+                tags=tags,
+                options=kwargs,
+            )
+            yield benchmark, output
+
+    def _get_benchmark_function(self, rows, columns):
+        return lambda: int(rows) * [int(columns) * [0]]
+
+
+@conbenchlegacy.runner.register_benchmark
+class CasesBenchmarkTypeCasting(conbenchlegacy.runner.Benchmark):
+    """Example benchmark with cases."""
+
+    name = "matrix-types"
+    valid_cases = (
+        ("rows", "columns", "flag"),
+        (10, 10.0, False),
+        (2, 2.0, False),
+        (10, 10.0, True),
+        (2, 2.0, True),
+    )
+
+    def run(self, case=None, **kwargs):
+        for case in self.get_cases(case, kwargs):
+            rows, columns, flag = case
+            tags = {"rows": rows, "columns": columns, "flag": flag}
+            func = self._get_benchmark_function(rows, columns, flag)
+            benchmark, output = self.conbench.benchmark(
+                func,
+                self.name,
+                tags=tags,
+                options=kwargs,
+            )
+            yield benchmark, output
+
+    def _get_benchmark_function(self, rows, columns, flag):
+        return lambda: rows * [int(columns) * [flag]]
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that just records external results."""
+
+    external = True
+    name = "external"
+
+    def run(self, **kwargs):
+        # external results from an API call, command line execution, etc
+        result = {
+            "data": [100, 200, 300],
+            "unit": "i/s",
+            "times": [0.100, 0.200, 0.300],
+            "time_unit": "s",
+        }
+
+        context = {"benchmark_language": "C++"}
+        yield self.conbench.record(
+            result, self.name, context=context, options=kwargs, output=result
+        )
+
+
+@conbenchlegacy.runner.register_benchmark
+class DataAndErrorBenchmark(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that supplies both data and an error."""
+
+    external = True
+    name = "data_and_error"
+
+    def run(self, **kwargs):
+        # external results from an API call, command line execution, etc
+        result = {
+            "data": [100, 200, 300],
+            "unit": "i/s",
+            "times": [0.100, 0.200, 0.300],
+            "time_unit": "s",
+        }
+
+        context = {"benchmark_language": "C++"}
+        yield self.conbench.record(
+            result,
+            self.name,
+            error={"something": "bad"},
+            context=context,
+            options=kwargs,
+            output=result,
+        )
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmarkR(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that records an R benchmark result."""
+
+    external = True
+    name = "external-r"
+
+    def run(self, **kwargs):
+        result, output = self._run_r_command()
+        info, context = self.conbench.get_r_info_and_context()
+
+        yield self.conbench.record(
+            {"data": [result], "unit": "s"},
+            self.name,
+            info=info,
+            context=context,
+            options=kwargs,
+            output=output,
+        )
+
+    def _run_r_command(self):
+        output, _ = self.conbench.execute_r_command(self._get_r_command())
+        result = float(output.split("\n")[-1].split("[1] ")[1])
+        return result, output
+
+    def _get_r_command(self):
+        return (
+            f"addition <- function() { 1 + 1 }; "
+            "start_time <- Sys.time();"
+            "addition(); "
+            "end_time <- Sys.time(); "
+            "result <- end_time - start_time; "
+            "as.numeric(result); "
+        )
+
+
+@conbenchlegacy.runner.register_benchmark
+class ExternalBenchmarkOptionsR(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that records an R benchmark result (with options)."""
+
+    external = True
+    name = "external-r-options"
+    options = {
+        "iterations": {"default": 1, "type": int},
+        "drop_caches": {"type": bool, "default": "false"},
+    }
+
+    def run(self, **kwargs):
+        data, iterations = [], kwargs.get("iterations", 1)
+        info, context = self.conbench.get_r_info_and_context()
+
+        for _ in range(iterations):
+            if kwargs.get("drop_caches", False):
+                self.conbench.sync_and_drop_caches()
+            result, output = self._run_r_command()
+            data.append(result["result"][0]["real"])
+
+        yield self.conbench.record(
+            {"data": data, "unit": "s"},
+            self.name,
+            info=info,
+            context=context,
+            options=kwargs,
+            output=output,
+        )
+
+    def _run_r_command(self):
+        r_command = self._get_r_command()
+        self.conbench.execute_r_command(r_command)
+        with open("placebo.json") as json_file:
+            data = json.load(json_file)
+        return data, json.dumps(data, indent=2)
+
+    def _get_r_command(self):
+        return (
+            "library(arrowbench); "
+            "out <- run_one(arrowbench:::placebo); "
+            "cat(jsonlite::toJSON(out), file='placebo.json'); "
+        )
+
+
+# Overriding Conbench so it is possible to test what benchmark results are published
+class TestConbench(conbenchlegacy.runner.Conbench):
+    def __init__(self):
+        self.published_benchmark = None
+        super().__init__()
+
+    def publish(self, benchmark):
+        self.published_benchmark = benchmark
+
+
+@conbenchlegacy.runner.register_benchmark
+class SimpleBenchmarkThatFails(conbenchlegacy.runner.Benchmark):
+    name = "division-with-failure"
+
+    def __init__(self):
+        self.conbench = TestConbench()
+
+    def run(self, **kwargs):
+        yield self.conbench.benchmark(
+            self._get_benchmark_function(), self.name, options=kwargs
+        )
+
+    def _get_benchmark_function(self):
+        return lambda: 1 / 0
+
+
+class ExternalBenchmarkWithWrongName(conbenchlegacy.runner.Benchmark):
+    """Example benchmark that uses a different name in tags than in record()."""
+
+    external = True
+    name = "some_name"
+
+    def run(self, **kwargs):
+        result = {
+            "data": [100, 200, 300],
+            "unit": "i/s",
+            "times": [0.100, 0.200, 0.300],
+            "time_unit": "s",
+        }
+        tags = {"name": "different_name"}
+
+        context = {"benchmark_language": "C++"}
+        yield self.conbench.record(
+            result, self.name, context=context, options=kwargs, output=result, tags=tags
+        )
diff --git a/legacy/tests/benchmark/test_cli.py b/legacy/tests/benchmark/test_cli.py
new file mode 100755
index 000000000..d429088a3
--- /dev/null
+++ b/legacy/tests/benchmark/test_cli.py
@@ -0,0 +1,395 @@
+import importlib.metadata as importlib_metadata
+import os
+import unittest.mock
+
+import pytest
+from conbenchlegacy.util import register_benchmarks
+
+__version__ = importlib_metadata.version("conbenchlegacy")
+
+pytestmark = pytest.mark.filterwarnings(
+    "ignore::conbenchlegacy.machine_info.GitParseWarning"
+)
+
+CONBENCH = """
+Usage: conbench [OPTIONS] COMMAND [ARGS]...
+
+  Conbench: Language-independent Continuous Benchmarking (CB) Framework
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  addition               Run addition benchmark.
+  data_and_error         Run data_and_error benchmark.
+  division-with-failure  Run division-with-failure benchmark.
+  external               Run external benchmark.
+  external-r             Run external-r benchmark.
+  external-r-options     Run external-r-options benchmark.
+  list                   List of benchmarks (for orchestration).
+  matrix                 Run matrix benchmark(s).
+  matrix-types           Run matrix-types benchmark(s).
+  product                Run product benchmark.
+  version                Display Conbench version.
+"""
+
+CONBENCH_LIST = """
+[
+  {
+    "command": "addition --iterations=2"
+  },
+  {
+    "command": "data_and_error --iterations=2"
+  },
+  {
+    "command": "division-with-failure --iterations=2"
+  },
+  {
+    "command": "external --iterations=2"
+  },
+  {
+    "command": "external-r --iterations=2"
+  },
+  {
+    "command": "external-r-options --iterations=2"
+  },
+  {
+    "command": "matrix --all=true --iterations=2"
+  },
+  {
+    "command": "matrix-types --all=true --iterations=2"
+  },
+  {
+    "command": "product --iterations=2"
+  }
+]
+"""
+
+
+CONBENCH_ADDITION = """
+Benchmark output:
+2
+"""
+
+
+CONBENCH_ADDITION_HELP = """
+Usage: conbench addition [OPTIONS]
+
+  Run addition benchmark.
+
+Options:
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --gc-collect BOOLEAN   [default: true]
+  --gc-disable BOOLEAN   [default: true]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+CONBENCH_MATRIX = """
+Benchmark output:
+[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
+"""
+
+CONBENCH_MATRIX_TYPES = """
+Benchmark output:
+[[False, False], [False, False]]
+"""
+
+CONBENCH_MATRIX_HELP = """
+Usage: conbench matrix [OPTIONS]
+
+  Run matrix benchmark(s).
+
+  For each benchmark option, the first option value is the default.
+
+  Valid benchmark combinations:
+  --rows=10 --columns=10
+  --rows=2 --columns=10
+  --rows=10 --columns=2
+
+  To run all combinations:
+  $ conbench matrix --all=true
+
+Options:
+  --rows [10|2]
+  --columns [10|2]
+  --all BOOLEAN          [default: false]
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --gc-collect BOOLEAN   [default: true]
+  --gc-disable BOOLEAN   [default: true]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+CONBENCH_MATRIX_HELP_TYPES = """
+Usage: conbench matrix-types [OPTIONS]
+
+  Run matrix-types benchmark(s).
+
+  For each benchmark option, the first option value is the default.
+
+  Valid benchmark combinations:
+  --rows=10 --columns=10.0 --flag=False
+  --rows=2 --columns=2.0 --flag=False
+  --rows=10 --columns=10.0 --flag=True
+  --rows=2 --columns=2.0 --flag=True
+
+  To run all combinations:
+  $ conbench matrix-types --all=true
+
+Options:
+  --rows [2|10]
+  --columns [2.0|10.0]
+  --flag [False|True]
+  --all BOOLEAN          [default: false]
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --gc-collect BOOLEAN   [default: true]
+  --gc-disable BOOLEAN   [default: true]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+CONBENCH_EXTERNAL = """
+Benchmark output:
+{'data': [100, 200, 300], 'unit': 'i/s', 'times': [0.1, 0.2, 0.3], 'time_unit': 's'}
+"""
+
+
+CONBENCH_EXTERNAL_HELP = """
+Usage: conbench external [OPTIONS]
+
+  Run external benchmark.
+
+Options:
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+CONBENCH_EXTERNAL_R_HELP = """
+Usage: conbench external-r [OPTIONS]
+
+  Run external-r benchmark.
+
+Options:
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+CONBENCH_EXTERNAL_R_OPTIONS_HELP = """
+Usage: conbench external-r-options [OPTIONS]
+
+  Run external-r-options benchmark.
+
+Options:
+  --iterations INTEGER   [default: 1]
+  --drop-caches BOOLEAN  [default: false]
+  --show-result BOOLEAN  [default: true]
+  --show-output BOOLEAN  [default: false]
+  --run-id TEXT          Group executions together with a run id.
+  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
+                         etc).
+  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
+                         manual, etc).
+  --help                 Show this message and exit.
+"""
+
+
+this_dir = os.path.dirname(os.path.abspath(__file__))
+register_benchmarks(this_dir)
+
+
+def assert_command_output(result, expected):
+    assert result.exit_code == 0
+    output = result.output.strip().replace("\x08", "")
+    assert output == expected.strip()
+
+
+def assert_command_contains(result, contains):
+    assert result.exit_code == 0
+    output = result.output.strip()
+    assert contains in output
+
+
+def test_conbench(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench)
+    assert_command_output(result, CONBENCH)
+
+
+def test_conbench_command_show_result(runner):
+    from conbenchlegacy.cli import conbench
+
+    command = "addition --show-result=true"
+    result = runner.invoke(conbench, command)
+    assert result.exit_code == 0
+    assert "tags" in result.output
+    assert "stats" in result.output
+    assert "context" in result.output
+    assert "machine_info" in result.output
+
+
+def test_conbench_list(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "list")
+    assert_command_output(result, CONBENCH_LIST)
+
+
+def test_conbench_command_without_cases(runner):
+    from conbenchlegacy.cli import conbench
+
+    command = "addition --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+    assert_command_output(result, CONBENCH_ADDITION)
+
+
+def test_conbench_command_without_cases_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "addition --help")
+    assert_command_output(result, CONBENCH_ADDITION_HELP)
+
+
+def test_conbench_command_with_cases(runner):
+    from conbenchlegacy.cli import conbench
+
+    case = "--rows=2 --columns=10"
+    command = f"matrix {case} --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+    assert_command_output(result, CONBENCH_MATRIX)
+
+
+def test_conbench_command_with_cases_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "matrix --help")
+    assert_command_output(result, CONBENCH_MATRIX_HELP)
+
+
+def test_conbench_command_with_cases_type_casting(runner):
+    from conbenchlegacy.cli import conbench
+
+    case = "--rows=2 --columns=2.0 --flag=false"
+    command = f"matrix-types {case} --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+    assert_command_output(result, CONBENCH_MATRIX_TYPES)
+
+
+def test_conbench_command_with_cases_type_casting_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "matrix-types --help")
+    assert_command_output(result, CONBENCH_MATRIX_HELP_TYPES)
+
+
+def test_conbench_command_external(runner):
+    from conbenchlegacy.cli import conbench
+
+    command = "external --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+    assert_command_output(result, CONBENCH_EXTERNAL)
+
+
+def test_conbench_command_external_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "external --help")
+    assert_command_output(result, CONBENCH_EXTERNAL_HELP)
+
+
+def test_conbench_command_external_r(runner):
+    from conbenchlegacy.cli import conbench
+    from conbenchlegacy.machine_info import r_info
+
+    try:
+        r_info()
+    except:
+        pytest.skip("No R")
+
+    command = "external-r --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+    assert_command_contains(result, "[1] 2")  # 1 + 1 = 2
+
+
+def test_conbench_command_external_r_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "external-r --help")
+    assert_command_output(result, CONBENCH_EXTERNAL_R_HELP)
+
+
+def test_conbench_command_external_options_r(runner):
+    from conbenchlegacy.cli import conbench
+    from conbenchlegacy.machine_info import r_info
+
+    try:
+        r_info()
+    except:
+        pytest.skip("No R")
+
+    command = "external-r-options --show-result=false --show-output=true"
+    with unittest.mock.patch("conbenchlegacy.util.Connection.publish"):
+        result = runner.invoke(conbench, command)
+
+    try:
+        assert_command_contains(result, '"result"')
+    except:
+        pytest.skip("Probably no arrowbench")
+
+
+def test_conbench_command_external_options_r_help(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "external-r-options --help")
+    assert_command_output(result, CONBENCH_EXTERNAL_R_OPTIONS_HELP)
+
+
+def test_conbench_command_version(runner):
+    from conbenchlegacy.cli import conbench
+
+    result = runner.invoke(conbench, "version")
+    assert_command_output(result, f"conbench version: {__version__}")
diff --git a/legacy/tests/benchmark/test_machine_info.py b/legacy/tests/benchmark/test_machine_info.py
new file mode 100644
index 000000000..097c04898
--- /dev/null
+++ b/legacy/tests/benchmark/test_machine_info.py
@@ -0,0 +1,353 @@
+from conbenchlegacy.machine_info import (
+    _fill_from_cpuinfo,
+    _fill_from_lscpu,
+    _fill_from_meminfo,
+    _round_memory,
+)
+
+MODEL_NAME = "Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz"
+L1D_CACHE = 524288
+L1I_CACHE = 524288
+L2_CACHE = 16777216
+L3_CACHE = 23068672
+MEMORY = 131590280000
+FREQUENCY = 4500000000
+
+
+LSCPU = [
+    "Architecture:                    x86_64",
+    "CPU op-mode(s):                  32-bit, 64-bit",
+    "Byte Order:                      Little Endian",
+    "Address sizes:                   46 bits physical, 48 bits virtual",
+    "CPU(s):                          32",
+    "On-line CPU(s) list:             0-31",
+    "Thread(s) per core:              2",
+    "Core(s) per socket:              16",
+    "Socket(s):                       1",
+    "NUMA node(s):                    1",
+    "Vendor ID:                       GenuineIntel",
+    "CPU family:                      6",
+    "Model:                           85",
+    "Model name:                      Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz",
+    "Stepping:                        4",
+    "CPU MHz:                         1577.989",
+    "CPU max MHz:                     4500.0000",
+    "CPU min MHz:                     1200.0000",
+    "BogoMIPS:                        6199.99",
+    "Virtualization:                  VT-x",
+    "L1d cache:                       524288",
+    "L1i cache:                       524288",
+    "L2 cache:                        16777216",
+    "L3 cache:                        23068672",
+    "NUMA node0 CPU(s):               0-31",
+    "Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled",
+    "Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable",
+    "Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable",
+    "Vulnerability Meltdown:          Mitigation; PTI",
+    "Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp",
+    "Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization",
+    "Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling",
+    "Vulnerability Srbds:             Not affected",
+    "Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable",
+    "Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d",
+]
+
+
+MEMINFO = [
+    "MemTotal:       131590280 kB",
+    "MemFree:        112655968 kB",
+    "MemAvailable:   129464936 kB",
+    "Buffers:          603140 kB",
+    "Cached:         16374620 kB",
+    "SwapCached:            0 kB",
+    "Active:         15203728 kB",
+    "Inactive:        2326196 kB",
+    "Active(anon):     312872 kB",
+    "Inactive(anon):     2352 kB",
+    "Active(file):   14890856 kB",
+    "Inactive(file):  2323844 kB",
+    "Unevictable:           0 kB",
+    "Mlocked:               0 kB",
+    "SwapTotal:       2097148 kB",
+    "SwapFree:        2097148 kB",
+    "Dirty:                 8 kB",
+    "Writeback:             0 kB",
+    "AnonPages:        552160 kB",
+    "Mapped:           221996 kB",
+    "Shmem:              4156 kB",
+    "KReclaimable:     848396 kB",
+    "Slab:            1187156 kB",
+    "SReclaimable:     848396 kB",
+    "SUnreclaim:       338760 kB",
+    "KernelStack:       12112 kB",
+    "PageTables:         9780 kB",
+    "NFS_Unstable:          0 kB",
+    "Bounce:                0 kB",
+    "WritebackTmp:          0 kB",
+    "CommitLimit:    67892288 kB",
+    "Committed_AS:    3204896 kB",
+    "VmallocTotal:   34359738367 kB",
+    "VmallocUsed:       66336 kB",
+    "VmallocChunk:          0 kB",
+    "Percpu:            25344 kB",
+    "HardwareCorrupted:     0 kB",
+    "AnonHugePages:    172032 kB",
+    "ShmemHugePages:        0 kB",
+    "ShmemPmdMapped:        0 kB",
+    "FileHugePages:         0 kB",
+    "FilePmdMapped:         0 kB",
+    "HugePages_Total:       0",
+    "HugePages_Free:        0",
+    "HugePages_Rsvd:        0",
+    "HugePages_Surp:        0",
+    "Hugepagesize:       2048 kB",
+    "Hugetlb:               0 kB",
+    "DirectMap4k:      566608 kB",
+    "DirectMap2M:    26359808 kB",
+    "DirectMap1G:    106954752 kB",
+]
+
+
+CPUINFO = {
+    "python_version": "3.8.5.final.0 (64 bit)",
+    "cpuinfo_version": [7, 0, 0],
+    "cpuinfo_version_string": "7.0.0",
+    "arch": "X86_64",
+    "bits": 64,
+    "count": 32,
+    "arch_string_raw": "x86_64",
+    "vendor_id_raw": "GenuineIntel",
+    "brand_raw": "Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz",
+    "hz_advertised_friendly": "3.1000 GHz",
+    "hz_actual_friendly": "1.7795 GHz",
+    "hz_advertised": [3100000000, 0],
+    "hz_actual": [1779488000, 0],
+    "stepping": 4,
+    "model": 85,
+    "family": 6,
+    "flags": [
+        "3dnowprefetch",
+        "abm",
+        "acpi",
+        "adx",
+        "aes",
+        "aperfmperf",
+        "apic",
+        "arat",
+        "arch_perfmon",
+        "art",
+        "avx",
+        "avx2",
+        "avx512bw",
+        "avx512cd",
+        "avx512dq",
+        "avx512f",
+        "avx512vl",
+        "bmi1",
+        "bmi2",
+        "bts",
+        "cat_l3",
+        "cdp_l3",
+        "clflush",
+        "clflushopt",
+        "clwb",
+        "cmov",
+        "constant_tsc",
+        "cpuid",
+        "cpuid_fault",
+        "cqm",
+        "cqm_llc",
+        "cqm_mbm_local",
+        "cqm_mbm_total",
+        "cqm_occup_llc",
+        "cx16",
+        "cx8",
+        "dca",
+        "de",
+        "ds_cpl",
+        "dtes64",
+        "dtherm",
+        "dts",
+        "epb",
+        "ept",
+        "ept_ad",
+        "erms",
+        "est",
+        "f16c",
+        "flexpriority",
+        "flush_l1d",
+        "fma",
+        "fpu",
+        "fsgsbase",
+        "fxsr",
+        "hle",
+        "ht",
+        "hwp",
+        "hwp_act_window",
+        "hwp_epp",
+        "hwp_pkg_req",
+        "ibpb",
+        "ibrs",
+        "ida",
+        "intel_pt",
+        "invpcid",
+        "invpcid_single",
+        "lahf_lm",
+        "lm",
+        "mba",
+        "mca",
+        "mce",
+        "md_clear",
+        "mmx",
+        "monitor",
+        "movbe",
+        "mpx",
+        "msr",
+        "mtrr",
+        "nonstop_tsc",
+        "nopl",
+        "nx",
+        "osxsave",
+        "pae",
+        "pat",
+        "pbe",
+        "pcid",
+        "pclmulqdq",
+        "pdcm",
+        "pdpe1gb",
+        "pebs",
+        "pge",
+        "pln",
+        "pni",
+        "popcnt",
+        "pqe",
+        "pqm",
+        "pse",
+        "pse36",
+        "pti",
+        "pts",
+        "rdrand",
+        "rdrnd",
+        "rdseed",
+        "rdt_a",
+        "rdtscp",
+        "rep_good",
+        "rtm",
+        "sdbg",
+        "sep",
+        "smap",
+        "smep",
+        "ss",
+        "ssbd",
+        "sse",
+        "sse2",
+        "sse4_1",
+        "sse4_2",
+        "ssse3",
+        "stibp",
+        "syscall",
+        "tm",
+        "tm2",
+        "tpr_shadow",
+        "tsc",
+        "tsc_adjust",
+        "tsc_deadline_timer",
+        "tscdeadline",
+        "vme",
+        "vmx",
+        "vnmi",
+        "vpid",
+        "x2apic",
+        "xgetbv1",
+        "xsave",
+        "xsavec",
+        "xsaveopt",
+        "xsaves",
+        "xtopology",
+        "xtpr",
+    ],
+    "l3_cache_size": 23068672,
+    "l2_cache_size": "16 MiB",
+    "l1_data_cache_size": "512 KiB",
+    "l1_instruction_cache_size": "512 KiB",
+    "l2_cache_line_size": 256,
+    "l2_cache_associativity": 6,
+}
+
+
+def test_machine_info_from_lscpu():
+    info = {
+        "cpu_model_name": "",
+        "cpu_l1d_cache_bytes": "",
+        "cpu_l1i_cache_bytes": "",
+        "cpu_l2_cache_bytes": "",
+        "cpu_l3_cache_bytes": "",
+        "cpu_frequency_max_hz": "",
+    }
+    _fill_from_lscpu(info, LSCPU)
+    assert info == {
+        "cpu_model_name": MODEL_NAME,
+        "cpu_l1d_cache_bytes": L1D_CACHE,
+        "cpu_l1i_cache_bytes": L1I_CACHE,
+        "cpu_l2_cache_bytes": L2_CACHE,
+        "cpu_l3_cache_bytes": L3_CACHE,
+        "cpu_frequency_max_hz": FREQUENCY,
+    }
+
+
+def test_machine_info_from_meminfo():
+    info = {"memory_bytes": ""}
+    _fill_from_meminfo(info, MEMINFO)
+    assert info == {"memory_bytes": MEMORY}
+
+
+def test_machine_info_from_cpuinfo():
+    info = {
+        "cpu_model_name": "",
+        "cpu_l1d_cache_bytes": "",
+        "cpu_l1i_cache_bytes": "",
+        "cpu_l2_cache_bytes": "",
+        "cpu_l3_cache_bytes": "",
+    }
+    _fill_from_cpuinfo(info, CPUINFO)
+    assert info == {
+        "cpu_model_name": MODEL_NAME,
+        "cpu_l1d_cache_bytes": L1D_CACHE,
+        "cpu_l1i_cache_bytes": L1I_CACHE,
+        "cpu_l2_cache_bytes": L2_CACHE,
+        "cpu_l3_cache_bytes": L3_CACHE,
+    }
+
+
+def test_fill_order():
+    info = {
+        "memory_bytes": "",
+        "cpu_model_name": "",
+        "cpu_l1d_cache_bytes": "",
+        "cpu_l1i_cache_bytes": "",
+        "cpu_l2_cache_bytes": "",
+        "cpu_l3_cache_bytes": "",
+        "cpu_frequency_max_hz": "",
+    }
+    _fill_from_meminfo(info, MEMINFO)
+    _fill_from_lscpu(info, LSCPU)
+    _fill_from_cpuinfo(info, CPUINFO)
+    assert info == {
+        "memory_bytes": MEMORY,
+        "cpu_model_name": MODEL_NAME,
+        "cpu_l1d_cache_bytes": L1D_CACHE,
+        "cpu_l1i_cache_bytes": L1I_CACHE,
+        "cpu_l2_cache_bytes": L2_CACHE,
+        "cpu_l3_cache_bytes": L3_CACHE,
+        "cpu_frequency_max_hz": FREQUENCY,
+    }
+
+
+def test_round_memory():
+    before = [131590280000, 131593068000, 131593872000]
+    after = [132070244352, 132070244352, 132070244352]
+    assert list(map(_round_memory, before)) == after
+
+    before = [15855024000, 15855036000, 15855040000]
+    after = [16106127360, 16106127360, 16106127360]
+    assert list(map(_round_memory, before)) == after
diff --git a/legacy/tests/benchmark/test_runner.py b/legacy/tests/benchmark/test_runner.py
new file mode 100644
index 000000000..2242449ce
--- /dev/null
+++ b/legacy/tests/benchmark/test_runner.py
@@ -0,0 +1,248 @@
+import copy
+import os
+
+import pytest
+
+from .. import _fixtures
+from ..helpers import _uuid
+from ._example_benchmarks import (
+    CasesBenchmark,
+    DataAndErrorBenchmark,
+    ExternalBenchmark,
+    ExternalBenchmarkWithWrongName,
+    SimpleBenchmark,
+    SimpleBenchmarkThatFails,
+    SimpleBenchmarkWithClusterInfo,
+)
+
+
+def validate_benchmark_result(result: dict) -> bool:
+    os.environ["CONBENCH_INTENDED_BASE_URL"] = "http://127.0.0.1:5000/"
+    from conbench.entities.benchmark_result import BenchmarkResultFacadeSchema
+
+    return BenchmarkResultFacadeSchema.create.validate(result)
+
+
+# this makes an implicit dependency become an explicit one, to save time
+# debugging test failures.
+assert os.path.exists(".git"), (
+    "expecting .git in current working directory, see "
+    "https://github.com/conbench/conbench/issues/668"
+)
+
+pytestmark = pytest.mark.filterwarnings(
+    "ignore::conbenchlegacy.machine_info.GitParseWarning"
+)
+
+EXAMPLE = copy.deepcopy(_fixtures.VALID_RESULT_PAYLOAD)
+EXAMPLE_WITH_CLUSTER_INFO = copy.deepcopy(_fixtures.VALID_RESULT_PAYLOAD_FOR_CLUSTER)
+EXAMPLE_WITH_ERROR = copy.deepcopy(_fixtures.VALID_RESULT_PAYLOAD_WITH_ERROR)
+for example in [EXAMPLE, EXAMPLE_WITH_CLUSTER_INFO, EXAMPLE_WITH_ERROR]:
+    example.pop("run_name")
+    example.pop("run_reason")
+    example.pop("validation")
+    example["info"] = {
+        "benchmark_language_version": "Python 3.8.5",
+    }
+    example["context"] = {
+        "benchmark_language": "Python",
+    }
+
+
+def assert_keys_equal(a, b):
+    assert set(a.keys()) == set(b.keys())
+
+
+def assert_repo_is_valid(repo):
+    assert repo.startswith("https://github.com/") or repo.startswith("git@github.com:")
+    assert repo.endswith("/conbench")
+
+
+def test_runner_simple_benchmark():
+    for benchmark_class, payload_example, hardware_type, tag in [
+        (SimpleBenchmark, EXAMPLE, "machine", "addition"),
+        (
+            SimpleBenchmarkWithClusterInfo,
+            EXAMPLE_WITH_CLUSTER_INFO,
+            "cluster",
+            "product",
+        ),
+    ]:
+        benchmark = benchmark_class()
+        [(result, output)] = benchmark.run(iterations=10)
+        assert not validate_benchmark_result(result)
+        expected_tags = {"name": tag}
+        assert output == 2
+        assert_keys_equal(result, payload_example)
+        assert_keys_equal(result["tags"], expected_tags)
+        assert_keys_equal(result["stats"], payload_example["stats"])
+        assert_keys_equal(result["info"], payload_example["info"])
+        assert_keys_equal(result["context"], payload_example["context"])
+        assert_keys_equal(result["github"], payload_example["github"])
+        assert_keys_equal(
+            result[f"{hardware_type}_info"], payload_example[f"{hardware_type}_info"]
+        )
+        assert result["tags"] == expected_tags
+        assert result["stats"]["iterations"] == 10
+        assert len(result["stats"]["data"]) == 10
+        assert result["context"]["benchmark_language"] == "Python"
+        assert_repo_is_valid(result["github"]["repository"])
+
+
+def test_runner_simple_benchmark_that_fails():
+    benchmark = SimpleBenchmarkThatFails()
+    tag = "division-with-failure"
+
+    with pytest.raises(ZeroDivisionError):
+        [(_, _)] = benchmark.run(iterations=10)
+
+    result = benchmark.conbench.published_benchmark
+    assert not validate_benchmark_result(result)
+    # print(result)
+    assert_keys_equal(result, EXAMPLE_WITH_ERROR)
+    for key in ["info", "context", "github", "machine_info"]:
+        assert_keys_equal(result[key], EXAMPLE_WITH_ERROR[key])
+    assert result["tags"] == {"name": tag}
+    assert result["context"]["benchmark_language"] == "Python"
+    assert_repo_is_valid(result["github"]["repository"])
+    assert "stats" not in result
+    assert "stack_trace" in result["error"]
+    for text in [
+        "Traceback (most recent call last):",
+        "data, output = self._get_timing(f, iterations, timing_options)",
+        "in _get_timing",
+        "output = f()",
+        "tests/benchmark/_example_benchmarks.py",
+        "return lambda: 1 / 0",
+        "ZeroDivisionError: division by zero",
+    ]:
+        assert text in result["error"]["stack_trace"]
+
+
+def test_runner_case_benchmark():
+    benchmark = CasesBenchmark()
+    case = ("2", "10")
+    [(result, output)] = benchmark.run(case=case, iterations=10)
+    assert not validate_benchmark_result(result)
+    expected_tags = {"name": "matrix", "rows": "2", "columns": "10"}
+    assert output == [
+        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+    ]
+    assert_keys_equal(result, EXAMPLE)
+    assert_keys_equal(result["tags"], expected_tags)
+    assert_keys_equal(result["stats"], EXAMPLE["stats"])
+    assert_keys_equal(result["info"], EXAMPLE["info"])
+    assert_keys_equal(result["context"], EXAMPLE["context"])
+    assert_keys_equal(result["github"], EXAMPLE["github"])
+    assert_keys_equal(result["machine_info"], EXAMPLE["machine_info"])
+    assert result["tags"] == expected_tags
+    assert result["stats"]["iterations"] == 10
+    assert len(result["stats"]["data"]) == 10
+    assert result["context"]["benchmark_language"] == "Python"
+    assert_repo_is_valid(result["github"]["repository"])
+
+
+def test_runner_external_benchmark():
+    benchmark = ExternalBenchmark()
+    [(result, output)] = benchmark.run()
+    assert not validate_benchmark_result(result)
+    expected_tags = {"name": "external"}
+    assert output == {
+        "data": [100, 200, 300],
+        "unit": "i/s",
+        "times": [0.1, 0.2, 0.3],
+        "time_unit": "s",
+    }
+    assert_keys_equal(result, EXAMPLE)
+    assert_keys_equal(result["tags"], expected_tags)
+    assert_keys_equal(result["stats"], EXAMPLE["stats"])
+    assert_keys_equal(result["github"], EXAMPLE["github"])
+    assert_keys_equal(result["machine_info"], EXAMPLE["machine_info"])
+    assert result["tags"] == expected_tags
+    assert result["stats"]["iterations"] == 3
+    assert len(result["stats"]["data"]) == 3
+    assert result["context"] == {"benchmark_language": "C++"}
+    assert_repo_is_valid(result["github"]["repository"])
+
+
+def test_runner_supplies_both_data_and_error():
+    benchmark = DataAndErrorBenchmark()
+    [(result, _)] = benchmark.run()
+    assert result["stats"]["iterations"] == 3
+    assert len(result["stats"]["data"]) == 3
+    assert result["error"] == {"something": "bad"}
+
+
+def test_runner_can_specify_run_and_batch_id():
+    benchmark = SimpleBenchmark()
+    run_id, batch_id = _uuid(), _uuid()
+    [(result, output)] = benchmark.run(run_id=run_id, batch_id=batch_id)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["run_id"] == run_id
+    assert result["batch_id"] == batch_id
+
+
+def test_runner_can_omit_run_and_batch_id():
+    benchmark = SimpleBenchmark()
+    [(result, output)] = benchmark.run()
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["run_id"] is not None
+    assert result["batch_id"] is not None
+
+
+def test_runner_null_run_and_batch_id():
+    benchmark = SimpleBenchmark()
+    [(result, output)] = benchmark.run(run_id=None, batch_id=None)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["run_id"] is not None
+    assert result["batch_id"] is not None
+
+
+def test_runner_omit_batch_id():
+    benchmark = SimpleBenchmark()
+    run_id = _uuid()
+    [(result, output)] = benchmark.run(run_id=run_id)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["run_id"] == run_id
+    assert result["batch_id"] is not None
+
+
+def test_runner_null_batch_id():
+    benchmark = SimpleBenchmark()
+    run_id = _uuid()
+    [(result, output)] = benchmark.run(run_id=run_id, batch_id=None)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["run_id"] == run_id
+    assert result["batch_id"] is not None
+
+
+def test_runner_omit_run_id():
+    benchmark = SimpleBenchmark()
+    batch_id = _uuid()
+    [(result, output)] = benchmark.run(batch_id=batch_id)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["batch_id"] == batch_id
+    assert result["run_id"] is not None
+
+
+def test_runner_null_run_id():
+    benchmark = SimpleBenchmark()
+    batch_id = _uuid()
+    [(result, output)] = benchmark.run(batch_id=batch_id, run_id=None)
+    assert not validate_benchmark_result(result)
+    assert output == 2
+    assert result["batch_id"] == batch_id
+    assert result["run_id"] is not None
+
+
+def test_runner_wrong_name():
+    benchmark = ExternalBenchmarkWithWrongName()
+    with pytest.raises(ValueError, match="do not match"):
+        next(benchmark.run())
diff --git a/legacy/tests/benchmark/test_util.py b/legacy/tests/benchmark/test_util.py
new file mode 100644
index 000000000..e59658df1
--- /dev/null
+++ b/legacy/tests/benchmark/test_util.py
@@ -0,0 +1,101 @@
+import tempfile
+
+from conbenchlegacy.util import Config, Connection, get_config
+
+CONFIG = b"""
+url: https://conbench.io
+email: cole@example.com
+password: woofwoof
+host_name: machine100
+"""
+
+
+def test_get_config():
+    with tempfile.NamedTemporaryFile(delete=False) as config_file:
+        config_file.write(CONFIG)
+        config_file.close()
+        config = get_config(config_file.name)
+        assert config == {
+            "email": "cole@example.com",
+            "password": "woofwoof",
+            "url": "https://conbench.io",
+            "host_name": "machine100",
+        }
+
+
+def test_default_config():
+    config = Config({})
+    assert config.login_url == "http://localhost:5000/api/login/"
+    assert config.benchmarks_url == "http://localhost:5000/api/benchmarks/"
+    credentials = {"email": "conbench@example.com", "password": "conbench"}
+    assert config.credentials == credentials
+    assert not config.host_name
+
+
+def test_custom_config():
+    custom = {
+        "email": "cole@example.com",
+        "password": "woofwoof",
+        "url": "https://conbench.io",
+        "host_name": "machine100",
+    }
+    config = Config(custom)
+    assert config.login_url == "https://conbench.io/api/login/"
+    assert config.benchmarks_url == "https://conbench.io/api/benchmarks/"
+    credentials = {"email": "cole@example.com", "password": "woofwoof"}
+    assert config.credentials == credentials
+    assert config.host_name == "machine100"
+
+
+EXPECTED_ERROR_JSON = """
+POST https://url.example.com failed
+{
+  "message": "some error message",
+  "code": "99"
+}
+
+"""
+
+EXPECTED_ERROR_HTML = """
+POST https://url.example.com failed
+<html>ERROR!</html>
+
+"""
+
+
+def test_unexpected_response_json(capsys):
+    class FakeResponse:
+        pass
+
+    url = "https://url.example.com"
+    method = "POST"
+    response = FakeResponse()
+    response.content = '{"message": "some error message", "code": "99"}'
+
+    connection = Connection()
+    connection.session = "something not None"
+    assert connection._unexpected_response(method, response, url) is None
+    assert connection.session is None
+
+    captured = capsys.readouterr()
+    assert captured.out == EXPECTED_ERROR_JSON
+    assert captured.err == ""
+
+
+def test_unexpected_response_not_html(capsys):
+    class FakeResponse:
+        pass
+
+    url = "https://url.example.com"
+    method = "POST"
+    response = FakeResponse()
+    response.content = "<html>ERROR!</html>"
+
+    connection = Connection()
+    connection.session = "something not None"
+    assert connection._unexpected_response(method, response, url) is None
+    assert connection.session is None
+
+    captured = capsys.readouterr()
+    assert captured.out == EXPECTED_ERROR_HTML
+    assert captured.err == ""
diff --git a/legacy/tests/conftest.py b/legacy/tests/conftest.py
new file mode 100644
index 000000000..e1add92eb
--- /dev/null
+++ b/legacy/tests/conftest.py
@@ -0,0 +1,7 @@
+import flask
+import pytest
+
+
+@pytest.fixture
+def runner():
+    return flask.Flask("test").test_cli_runner()
diff --git a/legacy/tests/helpers.py b/legacy/tests/helpers.py
new file mode 100644
index 000000000..ae90dc3ee
--- /dev/null
+++ b/legacy/tests/helpers.py
@@ -0,0 +1,5 @@
+import uuid
+
+
+def _uuid():
+    return uuid.uuid4().hex