Package: perf-baseline
Perf BaselineTheperf_baselineis a performance regression detection tool for Python projects.
It usestimeitto automatically time your function,
records the result as a baseline into a file,
and compares subsequent test results against the initial baseline
to detect performance regression based on your specified threshold.
(We do not compare against the second-last performance test result,
therefore your performance won't suffer from gradual decline.)Installationpip install perf_baselineUsageLet's say your project contains some important functions like this:defadd(a,b):returna+bdefsub(a,b):returna-bAnd you want to prevent your future refactoring
from accidentally introducing performance regression.
How? All you need to do is to create test cases like this:fromperf_baselineimportBaselinebaseline=Baseline("my_baseline.bin",# Performance test result will be saved into this filethreshold=1.8,# Subsequent tests will raise exception if a new test is more than 1.8x slower than the baseline)deftest_add_should_not_have_regression():baseline.set_or_compare("add(2, 3)",name="my addition implementation",globals={"add":add})deftest_sub_should_not_have_regression():baseline.set_or_compare("sub(5, 4)",globals={"sub":sub})# Note: When absent, name defaults to the current test function's name,#       which is "test_sub_should_not_have_regression" in this caseThat is it.Now you can runpytestto test it, orpytest --log-cli-level INFOto see some logs.
The test case will pass if there is no performance regression, or raiseRegressionErrorotherwise.Under the hood,perf_baselinestores theinitialtest results into the file you specified.
Each file may contain multiple data points, differentiated by their unique names.
If thenameparameter is omitted, the current test case's name will be used instead.Subsequent tests will automatically be compared with the initial baseline,
and error out when performance regression is detected.If you want to reset the baseline, simply delete that baseline file and then start afresh.What if the test subject requires nontrivial setup?In real world projects, your test subject may require nontrivial setup,
such as prepopulating some test data,
and those initialization time shall be excluded from benchmark.
How do we achieve that?A common solution is creating a wrapper class,
which has an expensive constructor and a (typically parameter-less) action method.
And then you can haveBaselineto check that action method.
For example:classTestDriver:def__init__(self,size):self.data=list(range(size))random.shuffle(self.data)defsort(self):# Some implementation to sort the self.data in-placebaseline=Baseline("my_baseline.bin",threshold=2.0)deftest_my_sort_implementation():driver=TestDriver(1000*1000)baseline.set_or_compare(driver.sort,# A parameter-less callable can be tested as-is, without setting globals)# You may also combine the above two lines into one,# and the initialization still runs only once.baseline.set_or_compare(TestDriver(1000*1000).sort,# The driver initialization is still done only once)Do NOT commit the baseline file into GitAdd your baseline filename into your.gitignore, so that you won't accidentally commit it.my_baseline.binWhy?The idea is that a performance baseline (such as 123456 ops/sec) is
only meaningful and consistent when running on thesamecomputer.
Switching to a different computer, it will have a different baseline.By not committing a baseline into the source code repo,
each maintainer of your project (and each of their computers)
will have their own baseline created by the first run.This way, you won't need to use a large threshold across different computers
(it is impossible to specify a constant threshold that works on different computers anyway).
Per-computer baselines all self-calibrate to match the performance of each computer.How to run this in Github Action?perf_baselinerelies on anupdatablebaseline file,
which shall be writable when a new data point (represented by a newname) occurs,
and remain read-only when an old data point (represented by samename) already exists.As of this writing,Github's Cache Actionsupports the updatable usage via some hack, inspired bythis hint.
Use the following snippet, and modify itspathandhashFiles(...)to match your filenames.-name:Setup an updatable cache for Performance Baselinesuses:actions/cache@v3with:path:my_baseline.binkey:${{ runner.os }}-performance-${{ hashFiles('tests/test_benchmark.py') }}restore-keys:${{ runner.os }}-performance-