Package: pytest-experiments
A pytest plugin to help developers of research-oriented software projects keep track of the results of their numerical experiments.What it doespytest-experimentsallowsresearch-oriented programmersto easily
persist data aboutnumerical experimentsso that they cantrack
metrics over time.Knowwhatexperiments you’ve run,whenyou ran them, itsinputsand itsresultsover time and project development.Review and compareexperiments over time to ensure that
continued development is improving your results.How it worksAnexperimentis a python function that runs your method or algorithm
against some input and reports one or more metrics of interest.Experiments are basicallyunit testsof numerical methods. Like unit tests
we provide a function or method under test with some input and assert that its
output conforms to some concrete expectations. Unlike unit tests, the method
under test produces some metrics which we are interested in but for which
concrete expectations do not exist. We store these metrics, along with some
metadata in a database so that we can track our results over time.We usepytestto collect and execute our experiments. This plugin offers
anotebookfixtureto facilitate recording your experiments. Here is
a very simple example experiment:importpytestfrommy_numerical_method_packageimport(my_implementation,# your numerical methodresult_is_valid,# returns True iff the result is well-formedperformance_metric,# the performance metric we care about)@pytest.mark.experiment# [optional] mark this test as an experiment@pytest.mark.parameterize("x",[1,2,3])# The inputs; we will run this experiment for x=1, x=2, and x=3deftest_my_numerical_method(notebook,x):# Request the notebook fixtureresult=my_implementation(x)assertresult_is_valid(result)# our concrete expectations about the resultnotebook.record(performance=performance_metric(result))# record the performanceAt the end of the test the notebook fixture will save experiment metadata, the
inputs, and whatever was passed tonotebook.recordto a database. By default,
this database will be a sqlite database calledexperiments.db.A machine learning exampleFor example, suppose we are building a machine learning classifier. The method
under test would be our model, the input would be train and validation
datasets and any hyper-parameters of our methods. The model is initialized
with the hyper-parameters, trained on the training data, and the output is the
predictions on the validation set.We want the model to return probabilities, so we have a concrete expectation
that the predictions should all be between 0 and 1. If any are not, our code
is wrong and the experiment should fail.However, we are not only interested in returning probabilities, we also want
our model to return good predictions (e.g. the predictions have high accuracy
and high fairness). We might have some conrete expectations about these metrics:
for example we may wish to reject any result that has metrics strictly worse
than some baseline, but it is not easy or meaningful to specify a criterion
based on the accuracy and fairness values for when we should stop developing
our model. In fact, the metrics collected during the experiment may inform
subsequent development.See thedemodirectory for a detailed example-based walkthrough.InstallationYou can install “pytest-experiments” viapipfromPyPI:$ pip install pytest-experimentsContributingContributions are very welcome. This project usespoetryfor packaging.To get set up simply clone the repo and runpoetry install
poetry run pre-commit installThe first command will install the package along with all development dependencies
in a virtual environment. The second command will install the pre-commit hook which
will automatically format source files withblack.Tests can be run withpytestPlease document any code added with docstrings. New modules can be auto-documented by
running:sphinx-apidoc -e -o docs/source src/pytest_experimentsDocumentation can be compiled (for example to html withmake):cd docs/
make htmlLicenseDistributed under the terms of theMITlicense, “pytest-experiments” is free and open source softwareIssuesIf you encounter any problems, pleasefile an issuealong with a detailed description.AcknowledgementsThispytestplugin was generated withCookiecutteralong with@hackebrot’scookiecutter-pytest-plugintemplate.