Package: conbench
ConbenchCheck out the docs athttps://conbench.github.io/conbench.Conbench allows you to write benchmarks in any language, publish the
results as JSON via an API, and persist them for comparison while
iterating on performance improvements or to guard against regressions.Conbench includes a runner which can be used as a stand-alone library
for traditional macro benchmark authoring. The runner will time a unit of
work (or measure throughput), collect machine information that may be relevant
for hardware specific optimizations, and return JSON formatted results.You can optionally host a Conbench server (API & dashboard) to share
benchmark results more widely, explore the changes over time, and
compare results across varying benchmark machines, languages, and cases.There is also a Conbench command line interface, useful for Continuous
Benchmarking (CB) orchestration alongside your development pipeline.TheApache Arrowproject is using Conbench
for Continuous Benchmarking. They have both native Python Conbench
benchmarks, and Conbench benchmarks written in Python that know how to
execute their external C++/R/Java/JavaScript benchmarks and record those results
too. Those benchmarks can be found in theursacomputing/benchmarksrepository, and the results are hosted on theArrow Conbench Server.May 2021:https://ursalabs.org/blog/announcing-conbench/InstallationAll packages in this repo can be installed from PyPI. Each package usesCalVerfor versioning. No stability is guaranteed between PyPI
versions, so consider pinning packages to a specific version in your code.pipinstallbenchadapt
pipinstallbenchalerts
pipinstallbenchclients
pipinstallbenchconnect
pipinstallbenchrun
pipinstallconbench# legacy CLIWe typically publish to PyPI often, when new features or bugfixes are needed by users,
but not on every merge tomain. To install the latest development version, install
from git like so:pipinstall'benchadapt@git+https://github.com/conbench/conbench.git@main#subdirectory=benchadapt/python'pipinstall'benchalerts@git+https://github.com/conbench/conbench.git@main#subdirectory=benchalerts'pipinstall'benchclients@git+https://github.com/conbench/conbench.git@main#subdirectory=benchclients/python'pipinstall'benchconnect@git+https://github.com/conbench/conbench.git@main#subdirectory=benchconnect'pipinstall'benchrun@git+https://github.com/conbench/conbench.git@main#subdirectory=benchrun/python'pipinstall'conbench@git+https://github.com/conbench/conbench.git@main'Developer environmentDependenciesmake,docker compose: common developer tasks depend on these tools. They need to be set up on your system.GITHUB_API_TOKENenvironment variable: set up a GitHub API token usingGitHub's instructions. It's recommended to only give the token read-only permissions to public repositories (which is the default for fine-grained personal access tokens). Runexport GITHUB_API_TOKEN="{token}"in your current shell.Makefile targetsThe following Makefile targets implement common developer tasks. They assume to be run in the root folder of the repository.make run-app: This command lets you experiment with Conbench locally.
It runs the stack in a containerized fashion.
It rebuilds container images from the current checkout, spawns
multiple containers (including one for the database), and then exposes Conbench's
HTTP server on the host athttp://127.0.0.1:5000.
The command will stay in the foreground of your terminal, showing log output of all containers.
Once you see access log lines likeGET /api/ping/ HTTP/1.1" 200you can point your browser tohttp://127.0.0.1:5000.
You can useCtrl+Cto terminate the containerized stack (this only stops containers, and the next invocation ofmake run-appwill use previous database state -- invokemake teardown-appto stop and remove containers).
If you wish to clear all database tables during local development you can hithttp://127.0.0.1:5000/api/wipe-dbwith the browser or with e.g. curl.make run-app-dev: Similar tomake run-app, but also mounts the repository's root directory into the container.
Code changes are (should be) detected automatically and result in automatic code reload.make tests: The nobrainer command to run the test suite just like CI does.
For more fine-grained control see further below.make lint: Performs invasive code linting in your local checkout.
May modify files. Analogue to what CI requires.
It requires for some commands to be available in your current shell.
Dependencies can be installed withpip install -r requirements-dev.txt.make alembic-new-migration: Attempts to generate a new migration Python module inmigrations/versions/.
Requires setting the environment variableALEMBIC_MIGRATION_NAMEbefore invocation.
Example:export ALEMBIC_MIGRATION_NAME='repo_url_lenth'.
After the file was created you may want to change its permissions and re-format it withblack.make conbench-on-minikube: requiresminikube.
Deploys the Conbench API server to a local minikube-powered Kubernetes cluster.
This also deploys a kube-prometheus-based observability stack.
Use this target for local development in this area.make docs-build: Builds HTML docs locally so you may check that they render
correctly with no linting problems. Dependencies can be installed withpip install -r requirements-dev.txt. Also, if you're working on the docstrings of
any of this repo's python packages, ensure the package is installed locally before
using this command.In CI, we usemake build-docs SPHINXOPTS='-W --keep-going'to fail the build if
there are Sphinx warnings. When using this command locally, you can just domake build-docs, but keep an eye on the warnings.View API documentationPoint your browser tohttp://127.0.0.1:5000/api/docs/.Python environment on the hostCI and common developer commands use containerized workflows where dependencies are defined and easy to reason about viaDockerfiles.Note that the CPython version that Conbench is tested with in CI and that it is recommended to be deployed with is currently the latest 3.11.x release, as also defined inDockerfileat the root of this repository.Some developer tasks may involve running Conbench tooling straight on the host.
Here is how to install the Python dependencies for the Conbench web application:pipinstall-rrequirements-webapp.txtDependencies for running code analysis and tests straight on the host can be installed withpipinstall-rrequirements-dev.txtDependencies for the (legacy)conbenchCLI can be installed withpipinstall-rrequirements-cli.txtFine-grained test invocationIfmake testis too coarse-grained, then this is how to take control of the containerizedpytesttest runner:dockercomposedown&&dockercomposebuildapp&&\dockercomposerunapp\pytest-vvconbench/testsThis command attempts to stop and remove previously spawned test runner containers, and it rebuilds theappcontainer image prior to running tests to pick up code changes in the local checkout.Useful command line arguments for local development (can be combined as desired):... pytest -k test_login: run only string-matching tests... pytest -x: exit upon first error... pytest -s: do not swallow log output during run... run -e CONBENCH_LOG_LEVEL_STDERR=DEBUG app ...Legacy commandsThe following commands are not guaranteed to work as documented, but provide valuable inspiration:To autogenerate a migration(conbench) $ brew services start postgres
(conbench) $ dropdb conbench_prod
(conbench) $ createdb conbench_prod
(conbench) $ git checkout main && git pull
(conbench) $ alembic upgrade head
(conbench) $ git checkout your-branch
(conbench) $ alembic revision --autogenerate -m "new"To populate local conbench with sample runs and benchmarksStart conbench app in Terminal window 1:(conbench) $ dropdb conbench_prod && createdb conbench_prod && alembic upgrade head && flask runRunconbench.tests.populate_local_conbenchin Terminal window 2 while conbench app is running:(conbench) $ python -m conbench.tests.populate_local_conbenchTo upload new version of packages to PyPIKick off a new run of the "Build and upload a package to PyPI" workflow on theActions
page.To add new documentation pagesTo add a new page to our GitHub Pages-hosted documentation:Add a Markdown file todocs/pages/.In the toctree in thedocs/index.rstfile, addpages/your_new_page, whereyour_new_pageis your new filename without the.mdfile suffix.To test that your new pages pass our documentation linter, run themake docs-buildcommand, as described above.Configuring the web applicationThe conbench web application can be configured with various environment variables as
defined inconfig.py.
Instructions are in that file.Creating accountsBy default, conbench has open read access, so a user account is not required to
view results or read from the API. An account is required only if the conbench
instance is private or to write data to conbench.If you do need an account, follow the login screen's "Sign Up" link, and use the
registration key specified in the server configuration above. If you are a user
of conbench, you may need to talk to your user administrator to get the
registration key. SSO can be configured to avoid requiring the registration key.If you have an account and need to create an additional account (say for a machine
user of the API) either repeat the process if you have the registration key, or if
you don't have the registration key (say if your account uses SSO), when logged in,
go to the gear menu / Users and use the "Add User" button to create a new account
without the registration key.Authoring benchmarksThere are three main types of benchmarks: "simple benchmarks" that time the
execution of a unit of work, "external benchmarks" that just record benchmark
results that were obtained from some other benchmarking tool, and "case
benchmarks" which benchmark a unit of work under different scenarios (cases).Included in this repository are contrived, minimal examples of these different
kinds of benchmarks to be used as templates for benchmark authoring. These
example benchmarks and their tests can be found here:_example_benchmarks.pytest_cli.pytest_runner.pyExample simple benchmarksA "simple benchmark" runs and records the execution time of a unit of work.Implementation details: Note that this benchmark extendsconbench.runner.Benchmark, implements the minimum requiredrun()method, and registers itself with the@conbench.runner.register_benchmarkdecorator.importconbench.runner@conbench.runner.register_benchmarkclassSimpleBenchmark(conbench.runner.Benchmark):name="addition"defrun(self,**kwargs):yieldself.conbench.benchmark(self._get_benchmark_function(),self.name,options=kwargs)def_get_benchmark_function(self):returnlambda:1+1Successfully registered benchmarks appear in theconbench --helplist.
For this benchmark to appear, the file must match the following paterns (documented in theutils.py:register_benchmarksfunction:benchmark*.py*benchmark.py*benchmarks.py(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench --help
Usage: conbench [OPTIONS] COMMAND [ARGS]...

  Conbench: Language-independent Continuous Benchmarking (CB) Framework

Options:
  --help  Show this message and exit.

Commands:
  addition            Run addition benchmark.
  external            Run external benchmark.
  external-r          Run external-r benchmark.
  external-r-options  Run external-r-options benchmark.
  list                List of benchmarks (for orchestration).
  matrix              Run matrix benchmark(s).
  version             Display Conbench version.Benchmarks can be run from command line within the directory where the
benchmarks are defined.Benchmark classes can also be imported and executed via therunmethod which
accepts the same arguments that appear in the command line help.(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench addition --help

Usage: conbench addition [OPTIONS]

  Run addition benchmark.

Options:
  --iterations INTEGER   [default: 1]
  --drop-caches BOOLEAN  [default: false]
  --gc-collect BOOLEAN   [default: true]
  --gc-disable BOOLEAN   [default: true]
  --show-result BOOLEAN  [default: true]
  --show-output BOOLEAN  [default: false]
  --run-id TEXT          Group executions together with a run id.
  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
                         etc).
  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
                         manual, etc).
  --help                 Show this message and exit.Example command line execution:(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench addition

Benchmark result:
{
    "batch_id": "c9db942c27db4359923eb08aa553beb7",
    "run_id": "f6c7d0b3b3f146f9b1ad297fc6e5776b",
    "timestamp": "2021-11-09T22:09:17.790397+00:00",
    "context": {
        "benchmark_language": "Python"
    },
    "github": {
        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
        "repository": "https://github.com/conbench/conbench"
    },
    "info": {
        "benchmark_language_version": "Python 3.9.7"
    },
    "machine_info": {
        "architecture_name": "arm64",
        "cpu_core_count": "8",
        "cpu_frequency_max_hz": "0",
        "cpu_l1d_cache_bytes": "65536",
        "cpu_l1i_cache_bytes": "131072",
        "cpu_l2_cache_bytes": "4194304",
        "cpu_l3_cache_bytes": "0",
        "cpu_model_name": "Apple M1",
        "cpu_thread_count": "8",
        "gpu_count": "0",
        "gpu_product_names": [],
        "kernel_name": "20.6.0",
        "memory_bytes": "17179869184",
        "name": "diana",
        "os_name": "macOS",
        "os_version": "11.5.2"
    },
    "stats": {
        "data": [
            "0.000001"
        ],
        "iqr": "0.000000",
        "iterations": 1,
        "max": "0.000001",
        "mean": "0.000001",
        "median": "0.000001",
        "min": "0.000001",
        "q1": "0.000001",
        "q3": "0.000001",
        "stdev": 0,
        "time_unit": "s",
        "times": [],
        "unit": "s"
    },
    "tags": {
        "name": "addition"
    }
}Example Python execution:(conbench) $ python
>>> import json
>>> from conbench.tests.benchmark import _example_benchmarks
>>> benchmark = _example_benchmarks.SimpleBenchmark()
>>> [(result, output)] = benchmark.run(iterations=10)
>>> output
2
>>> print(json.dumps(result, indent=2))
{
  "run_id": "dfe3a816ca9e451a9da7d940a974cb95",
  "batch_id": "0e869934b391424a8199c485dfbbc066",
  "timestamp": "2021-11-09T22:11:25.262330+00:00",
  "stats": {
    "data": [
      "0.000002",
      "0.000001",
      "0.000000",
      "0.000001",
      "0.000001",
      "0.000001",
      "0.000001",
      "0.000000",
      "0.000001",
      "0.000001"
    ],
    "times": [],
    "unit": "s",
    "time_unit": "s",
    "iterations": 10,
    "mean": "0.000001",
    "median": "0.000001",
    "min": "0.000000",
    "max": "0.000002",
    "stdev": "0.000001",
    "q1": "0.000001",
    "q3": "0.000001",
    "iqr": "0.000000"
  },
  "machine_info": {
    "name": "diana",
    "os_name": "macOS",
    "os_version": "11.5.2",
    "architecture_name": "arm64",
    "kernel_name": "20.6.0",
    "memory_bytes": "17179869184",
    "cpu_model_name": "Apple M1",
    "cpu_core_count": "8",
    "cpu_thread_count": "8",
    "cpu_l1d_cache_bytes": "65536",
    "cpu_l1i_cache_bytes": "131072",
    "cpu_l2_cache_bytes": "4194304",
    "cpu_l3_cache_bytes": "0",
    "cpu_frequency_max_hz": "0",
    "gpu_count": "0",
    "gpu_product_names": []
  },
  "context": {
    "benchmark_language": "Python"
  },
  "info": {
    "benchmark_language_version": "Python 3.9.7"
  },
  "tags": {
    "name": "addition"
  },
  "github": {
    "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
    "repository": "https://github.com/conbench/conbench"
  }
}By default, Conbench will try to publish your results to a Conbench server. If
you don't have one running or are missing a.conbenchcredentials file, you'll
see error messages like the following when you execute benchmarks.POST http://localhost:5000/api/login/ failed
{
  "code": 400,
  "description": {
    "_errors": [
      "Invalid email or password."
    ]
  },
  "name": "Bad Request"
}

POST http://localhost:5000/api/benchmarks/ failed
{
  "code": 401,
  "name": "Unauthorized"
}To publish your results to a Conbench server, place a.conbenchfile in the
same directory as your benchmarks. Thecatcommand below shows the contents
of an example.conbenchconfig file.(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ cat .conbench
url: http://localhost:5000
email: you@example.com
password: conbenchIf you don't yet have a Conbench server user account, you'll need to create one
to publish results (registration key defaults toconbench).http://localhost:5000/register/Example external benchmarksAn "external benchmark" records results that were obtained from some other
benchmarking tool (like executing an R benchmark from command line, parsing
the resulting JSON, and recording those results).Implementation details: Note that the following benchmark setsexternal = True, and callsself.conbench.record()rather thanself.conbench.benchmark()as the example above does.importconbench.runner@conbench.runner.register_benchmarkclassExternalBenchmark(conbench.runner.Benchmark):"""Example benchmark that just records external results."""external=Truename="external"defrun(self,**kwargs):# external results from an API call, command line execution, etcresult={"data":[100,200,300],"unit":"i/s","times":[0.100,0.200,0.300],"time_unit":"s",}context={"benchmark_language":"C++"}yieldself.conbench.record(result,self.name,context=context,options=kwargs,output=result)(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench external --help

Usage: conbench external [OPTIONS]

  Run external benchmark.

Options:
  --show-result BOOLEAN  [default: true]
  --show-output BOOLEAN  [default: false]
  --run-id TEXT          Group executions together with a run id.
  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
                         etc).
  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
                         manual, etc).
  --help                 Show this message and exit.Note that the use of--iterations=3results in 3 runs of the benchmark, and
themean,stdev, etc calculated.(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench external --iterations=3

Benchmark result:
{
    "run_id": "8058dde1491b49e5bd514646797c2a20",
    "batch_id": "8058dde1491b49e5bd514646797c2a20",
    "timestamp": "2021-06-21T22:16:54.786499+00:00",
    "context": {
        "benchmark_language": "C++"
    },
    "github": {
        "commit": "58fb35dc593dca82c917cf18c1c65c059b9fb12c",
        "repository": "https://github.com/conbench/conbench"
    },
    "info": {},
    "machine_info": {
        "architecture_name": "x86_64",
        "cpu_core_count": "2",
        "cpu_frequency_max_hz": "3500000000",
        "cpu_l1d_cache_bytes": "32768",
        "cpu_l1i_cache_bytes": "32768",
        "cpu_l2_cache_bytes": "262144",
        "cpu_l3_cache_bytes": "4194304",
        "cpu_model_name": "Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz",
        "cpu_thread_count": "4",
        "kernel_name": "20.5.0",
        "memory_bytes": "17179869184",
        "name": "machine-abc",
        "os_name": "macOS",
        "os_version": "10.16"
    },
    "stats": {
        "data": [
            "100.000000",
            "200.000000",
            "300.000000"
        ],
        "iqr": "100.000000",
        "iterations": 3,
        "max": "300.000000",
        "mean": "200.000000",
        "median": "200.000000",
        "min": "100.000000",
        "q1": "150.000000",
        "q3": "250.000000",
        "stdev": "100.000000",
        "time_unit": "s",
        "times": [
            "0.100000",
            "0.200000",
            "0.300000"
        ],
        "unit": "i/s"
    },
    "tags": {
        "name": "external"
    }
}Example simple benchmarks executed on machine cluster instead of one machineIf your benchmark is executed on a machine cluster instead of one machine, you
can capture cluster info in the following manner.
Note that a benchmark will have a continuous history on a specific cluster as long as cluster'snameandinfodo not change.
There is also anoptional_infofield for information that should not impact the cluster's hash (and thus disrupt the distribution history), but should still be recorded.importconbench.runner@conbench.runner.register_benchmarkclassSimpleBenchmarkWithClusterInfo(conbench.runner.Benchmark):name="product"defrun(self,**kwargs):cluster_info={"name":"cluster 1","info":{"gpu":1},"optional_info":{"workers":2},}yieldself.conbench.benchmark(self._get_benchmark_function(),self.name,cluster_info=cluster_info,options=kwargs,)def_get_benchmark_function(self):returnlambda:1*2Example case benchmarksA "case benchmark" is a either a "simple benchmark" or an "external benchmark"
executed under various predefined scenarios (cases).Implementation details: Note that the following benchmark declares the valid
combinations invalid_cases, which reads like a CSV (the first row contains
the cases names).importconbench.runner@conbench.runner.register_benchmarkclassCasesBenchmark(conbench.runner.Benchmark):"""Example benchmark with cases."""name="matrix"valid_cases=(("rows","columns"),("10","10"),("2","10"),("10","2"),)defrun(self,case=None,**kwargs):forcaseinself.get_cases(case,kwargs):rows,columns=casetags={"rows":rows,"columns":columns}func=self._get_benchmark_function(rows,columns)benchmark,output=self.conbench.benchmark(func,self.name,tags=tags,options=kwargs,)yieldbenchmark,outputdef_get_benchmark_function(self,rows,columns):returnlambda:int(rows)*[int(columns)*[0]](conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench matrix --help

Usage: conbench matrix [OPTIONS]

  Run matrix benchmark(s).

  For each benchmark option, the first option value is the default.

  Valid benchmark combinations:
  --rows=10 --columns=10
  --rows=2 --columns=10
  --rows=10 --columns=2

  To run all combinations:
  $ conbench matrix --all=true

Options:
  --rows [10|2]
  --columns [10|2]
  --all BOOLEAN          [default: false]
  --iterations INTEGER   [default: 1]
  --drop-caches BOOLEAN  [default: false]
  --gc-collect BOOLEAN   [default: true]
  --gc-disable BOOLEAN   [default: true]
  --show-result BOOLEAN  [default: true]
  --show-output BOOLEAN  [default: false]
  --run-id TEXT          Group executions together with a run id.
  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
                         etc).
  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
                         manual, etc).
  --help                 Show this message and exit.
    """Note that the use of--all=trueresults in 3 benchmark results, one for each
case (10 x 10,2, x 10, and10, x 2).(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench matrix --all=true

Benchmark result:
{
    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
    "run_id": "48acd853b8294df9a1f5457f192456f3",
    "timestamp": "2021-11-09T22:15:23.501923+00:00",
    "context": {
        "benchmark_language": "Python"
    },
    "github": {
        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
        "repository": "https://github.com/conbench/conbench"
    },
    "info": {
        "benchmark_language_version": "Python 3.9.7"
    },
    "machine_info": {
        "architecture_name": "arm64",
        "cpu_core_count": "8",
        "cpu_frequency_max_hz": "0",
        "cpu_l1d_cache_bytes": "65536",
        "cpu_l1i_cache_bytes": "131072",
        "cpu_l2_cache_bytes": "4194304",
        "cpu_l3_cache_bytes": "0",
        "cpu_model_name": "Apple M1",
        "cpu_thread_count": "8",
        "gpu_count": "0",
        "gpu_product_names": [],
        "kernel_name": "20.6.0",
        "memory_bytes": "17179869184",
        "name": "diana",
        "os_name": "macOS",
        "os_version": "11.5.2"
    },
    "run_id": "48acd853b8294df9a1f5457f192456f3",
    "stats": {
        "data": [
            "0.000004"
        ],
        "iqr": "0.000000",
        "iterations": 1,
        "max": "0.000004",
        "mean": "0.000004",
        "median": "0.000004",
        "min": "0.000004",
        "q1": "0.000004",
        "q3": "0.000004",
        "stdev": 0,
        "time_unit": "s",
        "times": [],
        "unit": "s"
    },
    "tags": {
        "columns": "10",
        "name": "matrix",
        "rows": "10"
    },
    "timestamp": "2021-11-09T22:15:23.397819+00:00"
}

Benchmark result:
{
    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
    "context": {
        "benchmark_language": "Python"
    },
    "github": {
        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
        "repository": "https://github.com/conbench/conbench"
    },
    "info": {
        "benchmark_language_version": "Python 3.9.7"
    },
    "machine_info": {
        "architecture_name": "arm64",
        "cpu_core_count": "8",
        "cpu_frequency_max_hz": "0",
        "cpu_l1d_cache_bytes": "65536",
        "cpu_l1i_cache_bytes": "131072",
        "cpu_l2_cache_bytes": "4194304",
        "cpu_l3_cache_bytes": "0",
        "cpu_model_name": "Apple M1",
        "cpu_thread_count": "8",
        "gpu_count": "0",
        "gpu_product_names": [],
        "kernel_name": "20.6.0",
        "memory_bytes": "17179869184",
        "name": "diana",
        "os_name": "macOS",
        "os_version": "11.5.2"
    },
    "stats": {
        "data": [
            "0.000004"
        ],
        "iqr": "0.000000",
        "iterations": 1,
        "max": "0.000004",
        "mean": "0.000004",
        "median": "0.000004",
        "min": "0.000004",
        "q1": "0.000004",
        "q3": "0.000004",
        "stdev": 0,
        "time_unit": "s",
        "times": [],
        "unit": "s"
    },
    "tags": {
        "columns": "10",
        "name": "matrix",
        "rows": "2"
    }
}

Benchmark result:
{
    "batch_id": "13b87cc6d9a84f2188df279d8c513933",
    "run_id": "48acd853b8294df9a1f5457f192456f3",
    "timestamp": "2021-11-09T22:15:23.509211+00:00",
    "context": {
        "benchmark_language": "Python"
    },
    "github": {
        "commit": "61dec915b9dd230ca5029f5e586f8bd95c3e0c05",
        "repository": "https://github.com/conbench/conbench"
    },
    "info": {
        "benchmark_language_version": "Python 3.9.7"
    },
    "machine_info": {
        "architecture_name": "arm64",
        "cpu_core_count": "8",
        "cpu_frequency_max_hz": "0",
        "cpu_l1d_cache_bytes": "65536",
        "cpu_l1i_cache_bytes": "131072",
        "cpu_l2_cache_bytes": "4194304",
        "cpu_l3_cache_bytes": "0",
        "cpu_model_name": "Apple M1",
        "cpu_thread_count": "8",
        "gpu_count": "0",
        "gpu_product_names": [],
        "kernel_name": "20.6.0",
        "memory_bytes": "17179869184",
        "name": "diana",
        "os_name": "macOS",
        "os_version": "11.5.2"
    },
    "stats": {
        "data": [
            "0.000002"
        ],
        "iqr": "0.000000",
        "iterations": 1,
        "max": "0.000002",
        "mean": "0.000002",
        "median": "0.000002",
        "min": "0.000002",
        "q1": "0.000002",
        "q3": "0.000002",
        "stdev": 0,
        "time_unit": "s",
        "times": [],
        "unit": "s"
    },
    "tags": {
        "columns": "2",
        "name": "matrix",
        "rows": "10"
    }
}Example R benchmarksHere are a few examples illustrating how to integrate R benchmarks with
Conbench.The first one just times1 + 1in R, and the second one executes an R
benchmark from a library of R benchmarks (in this casearrowbench).If you find yourself wrapping a lot of R benchmarks in Python to integrate them
with Conbench (to get uniform JSON benchmark results which you can persist and
publish on a Conbench server), you'll probably want to extract much of the
boilerplate out into a base class.importconbench.runner@conbench.runner.register_benchmarkclassExternalBenchmarkR(conbench.runner.Benchmark):"""Example benchmark that records an R benchmark result."""external=Truename="external-r"defrun(self,**kwargs):result,output=self._run_r_command()info,context=self.conbench.get_r_info_and_context()yieldself.conbench.record({"data":[result],"unit":"s"},self.name,info=info,context=context,options=kwargs,output=output,)def_run_r_command(self):output,_=self.conbench.execute_r_command(self._get_r_command())result=float(output.split("\n")[-1].split("[1] ")[1])returnresult,outputdef_get_r_command(self):return(f"addition <- function(){1+1}; ""start_time <- Sys.time();""addition(); ""end_time <- Sys.time(); ""result <- end_time - start_time; ""as.numeric(result); ")(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench external-r --help

Usage: conbench external-r [OPTIONS]

  Run external-r benchmark.

Options:
  --show-result BOOLEAN  [default: true]
  --show-output BOOLEAN  [default: false]
  --run-id TEXT          Group executions together with a run id.
  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
                         etc).
  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
                         manual, etc).
  --help                 Show this message and exit.importjsonimportconbench.runner@conbench.runner.register_benchmarkclassExternalBenchmarkOptionsR(conbench.runner.Benchmark):"""Example benchmark that records an R benchmark result (with options)."""external=Truename="external-r-options"options={"iterations":{"default":1,"type":int},"drop_caches":{"type":bool,"default":"false"},}defrun(self,**kwargs):data,iterations=[],kwargs.get("iterations",1)info,context=self.conbench.get_r_info_and_context()for_inrange(iterations):ifkwargs.get("drop_caches",False):self.conbench.sync_and_drop_caches()result,output=self._run_r_command()data.append(result["result"][0]["real"])yieldself.conbench.record({"data":data,"unit":"s"},self.name,info=info,context=context,options=kwargs,output=output,)def_run_r_command(self):r_command=self._get_r_command()self.conbench.execute_r_command(r_command)withopen("placebo.json")asjson_file:data=json.load(json_file)returndata,json.dumps(data,indent=2)def_get_r_command(self):return("library(arrowbench); ""out <- run_one(arrowbench:::placebo); ""cat(jsonlite::toJSON(out), file='placebo.json'); ")(conbench) $ cd ~/workspace/conbench/conbench/tests/benchmark/
(conbench) $ conbench external-r --help

Usage: conbench external-r-options [OPTIONS]

  Run external-r-options benchmark.

Options:
  --iterations INTEGER   [default: 1]
  --drop-caches BOOLEAN  [default: false]
  --show-result BOOLEAN  [default: true]
  --show-output BOOLEAN  [default: false]
  --run-id TEXT          Group executions together with a run id.
  --run-name TEXT        Free-text name of run (commit ABC, pull request 123,
                         etc).
  --run-reason TEXT      Low-cardinality reason for run (commit, pull request,
                         manual, etc).
  --help                 Show this message and exit.