Package: benchrun
Python {benchrun}{benchrun} is a Python package to run macrobenchmarks, deliberately designed to work
well with the larger conbench ecosystem.Installation{benchrun} is not [yet] on a package archive like PyPI; you can install from GitHub
withpipinstallbenchrun@git+https://github.com/conbench/conbench.git@main#subdirectory=benchrun/pythonWriting benchmarksIterationThe code to run for a benchmark is contained in a class inheriting from the abstractIterationclass. At a minimum, users must override thenameattribute andrun()method (the code to time), but may also overridesetup(),before_each(),after_each()andteardown()methods, where*_each()runs before/after each
iteration, andsetup()andteardown()run once before/after all iterations. A
simple implementation might look likeimporttimefrombenchrunimportIterationclassMyIteration(Iteration):name="my-iteration"defbefore_each(self,case:dict)->None:# use the `env` dict attribute to pass data between stagesself.env={"success":False}defrun(self,case:dict)->dict:# code to time goes heretime.sleep(case["sleep_seconds"])self.env["success"]=Truedefafter_each(self,case:dict)->None:assertrun_results["success"]self.env={}CaseListAnIteration's methods are parameterized withcase, a dict where keys are
parameters for the benchmark, and the values are scalar arguments. Cases are managed
with an instance ofCaseList, a class which takes aparamsdict, which is like a
case dict with the difference that the arguments are lists of valid arguments, not
scalars.CaseListwill populate acase_listattribute which contains the grid of
specified cases to be run:frombenchrunimportCaseListcase_list=CaseList(params={"x":[1,2],"y":["a","b","c"]})case_list.case_list#> [{'x': 1, 'y': 'a'},#>  {'x': 1, 'y': 'b'},#>  {'x': 1, 'y': 'c'},#>  {'x': 2, 'y': 'a'},#>  {'x': 2, 'y': 'b'},#>  {'x': 2, 'y': 'c'}]CaseListcontains an overridablefilter_cases()method that can be used to remove
invalid combinations of parameters, e.g. if anxof2with ayofbis not
viable:classMyCaseList(CaseList):deffilter_cases(self,case_list:List[Dict[str,Any]])->List[Dict[str,Any]]:filtered_case_list=[]forcaseincase_list:ifnot(case["x"]==2andcase["y"]=="b"):filtered_case_list.append(case)returnfiltered_case_listmy_case_list=MyCaseList(params={"x":[1,2],"y":["a","b","c"]})my_case_list.case_list#> [{'x': 1, 'y': 'a'},#>  {'x': 1, 'y': 'b'},#>  {'x': 1, 'y': 'c'},#>  {'x': 2, 'y': 'a'},#>  {'x': 2, 'y': 'c'}]If there are so many restrictions that it is simpler to specify which cases are
viable than which are not, thecase_listparameter offilter_cases()can be
completely ignored and a manually-generated list can be returned.BenchmarkABenchmarkin {benchrun} consists of anIterationinstance, aCaseListinstance, and potentially a bit more metadata about how to run it like whether to
drop disk caches beforehand.my_benchmark=Benchmark(iteration=my_iteration,case_list=my_case_list)This class has arun()method to run all cases, orrun_case()to run a single
case.BenchmarkListABenchmarkListis a lightweight class to tie together all the instances ofBenchmarkthat should be run together (e.g. all the benchmarks for a package).frombenchrunimportBenchmarkListmy_benchmark_list=BenchmarkList(benchmarks=[my_benchmark])The class has a__call__()method that will run all benchmarks in its list,
taking care that they all use the samerun_idso they will all appear together
on conbench.Running benchmarks and sending results to conbenchBenchmarkListis designed to work seamlessly with
{benchadapt}'sCallableAdapterclass:frombenchadapt.adaptersimportCallableAdaptermy_adapter=CallableAdapter(callable=my_benchmark_list)Like all adapters, it then has arun()method to run all the benchmarks it
contains (handling generic metadata appropriately for you), apost_results()method that will send the results to a conbench server, and a__call__()method
that will do both. These are the methods that should be called in whatever CI or
automated build system will be used for running benchmarks.Setting more metadata{benchrun} and {benchadapt} make an effort to handle as much metadata for you as
possible (e.g. things like machine info), but you will still need to specify some
metadata yourself, e.g. build flags used in compilation or things likerun_reason(often something likecommitormerge). To see what actually gets sent to
conbench, see the documentation forbenchadapt.BenchmarkResult.