Package: benchadapt
Python {benchadapt}A small python package with utilities for getting benchmark results into a Conbench server.Useful components in this packageBenchmarkResultandBenchmarkRundataclassesTheBenchmarkResultandBenchmarkRundataclasses are designed to make it easy to
populate JSON payloads to post to a Conbench server. Their structure corresponds to the
corresponding POST endpoint; they each have a.to_publishable_dict()method that
produces a dict to post.Regardless of how you are using them, the docstrings of these two objects will be useful
as you try to assemble your results to get them in Conbench. All fields are documented,
as are interactions between them and what you likely need to specify.The objects try to help you fill in your payloads correctly, including some defaults,
like populatingmachine_infowith metadata on the current machine. If you are running
on a cluster instead, you will need to populatecluster_infoyourself, andmachine_infowill remain empty.There is light validation, but [for now] the API is the ultimate validator; it is
possible to make payloads that will be rejected.If you need to interact directly with a Conbench webapp's API instead of letting adapters
(see below) or another tool manage sending results for you, you can usebenchclients.ConbenchClientto make requests. As benchclients is a dependency of benchadapt, you should not need to
install anything new, and it is nicely set up to handle auth and such for you.AdaptersThe concept of Conbench adapters is for when you already have a benchmarking method that
produces a pile of results (say in JSON files, though anything works), and you need to
transform them into an appropriate form that can be posted to a Conbench API.Thebenchadapt.adapters.BenchmarkAdapterabstract class defines a basic workflow:Call an arbitrarycommandshell command, presumably to run benchmarks. If results
are already guaranteed to exist, this can be set to do nothing.Transform results produced by the benchmarks into a list ofBenchmarkResultinstances.Postprocess results to ensure a consistentrun_idand override any metadata fields
not already set correctly.Post each result to a Conbench API.Classes that inherit from the abstract class need to defineHow to get results, including whatcommandshould be (though it can be defined later
by the user, if desired) and how to get the raw results (e.g. if they are in a file or
directory of files, where they are and how to read them in).How to transform the results into a list ofBenchmarkResultinstances ((2) above) in
the._transform_results()method.(3) and (4) are handled by the abstract class.Various adapters are alrady defined in theadapterssubmodule, including ones for
Google Benchmark and Folly, as well as a genericCallableAdapter, which takes a PythonCallableinstance (a function or class with a__call__()method) that returns a list
ofBenchmarkResultinstances directly instead of a shell command. Many more adapters
are possible; if you create one corresponding to a benchmarking tool, please make a PR!Running an adapterAdapters have separate.run()and.publish_results()methods; the former runs the
benchmarks, transforms the results, and stores them in a.resultsattribute of the
instance. It does not post them, so is useful for looking at results interactively before
sending them..publish_results()takes the results from the.resultsattribute and
posts them to a Conbench API.The whole instance also has a__call__()method defined so it can be called like a
function that both runs and publishes, so a somewhat minimal script for running
benchmarks in CI might look likeimportosfrombenchadapt.adaptersimportGoogleBenchmarkAdapteradapter=GoogleBenchmarkAdapter(command=["bash","./run-benchmarks.sh"],result_file="benchmarks.json",result_fields_override={"run_reason":os.getenv("CONBENCH_RUN_REASON")},result_fields_append={"info":{"build_version":os.getenv("MY_BUILD_VERSION")},"context":{"compiler_flags":os.getenv("MY_COMPILER_FLAGS")}})adapter()Of note:result_fields_overridewill replace the whole attribute with a new value. This works
with all types (strings, dicts, etc.), so hererun_reasonwill be set for all results.result_fields_appendwill append the new values to dicts which may already have data.
Here,build_versionwill be appended to theinfodict. In this case it is an empty
dict anyway, so this is equivalent toresult_fields_override={"info": {"build_version": os.getenv("MY_BUILD_VERSION")}}).
But thecontextdict will already contain a"benchmark_language"key; this will be
retained, andcompiler_flagswill be appended.For this to work, a lot of environment variables have to be set! This includes ones
with information about the Conbench server and the current git metadata. See the
"Environment Variables" section below for a full list.Environment variablesSome operations of benchadapt rely on a number of environment variables. The Conbench API
ones (CONBENCH_*) are used bybenchclients.ConbenchClient; the git ones
(CONBENCH_PROJECT_*) are used to populate run and result metadata if not specified
directly; andCONBENCH_MACHINE_INFO_NAMEis for overriding the machine name in
automatically gathered machine info when necessary:CONBENCH_URL: Required. The URL of the Conbench API without a trailing
slash, e.g.https://conbench.example.comCONBENCH_EMAIL: The email to use for Conbench loginCONBENCH_PASSWORD: The password to use for Conbench loginCONBENCH_PROJECT_REPOSITORY: The repository name (in the formatorg/repo) or the
URL (in the formathttps://github.com/org/repo)CONBENCH_PROJECT_PR_NUMBER: [recommended] The number of the GitHub pull request that
is running this benchmark. Do not supply this for a runs on the default branch.CONBENCH_PROJECT_COMMIT: The 40-character commit SHA of the repo being benchmarkedCONBENCH_MACHINE_INFO_NAME: By default, the running machine host name (sent inmachine_info.namewhen posting runs and benchmarks) will be obtained withplatform.node(), but in circumstances where consistency is needed (e.g.
running in CI or on cloud runners), a value for host name can be specified via
this environment variable instead.