Package: coiled-runtime
Coiled RuntimeThe Coiled Runtime is a conda metapackage which makes it easy to get started with Dask.Installcoiled-runtimecan be installed withconda:condainstall-cconda-forgecoiled-runtimeor withpip:pipinstallcoiled-runtimeNightly buildscoiled-runtimehas nightly conda packages for testing purposes.
You can install a nightly version ofcoiled-runtimewith:condainstall-ccoiled/label/dev-cdask/label/devcoiled-runtimeBuildTo build and installcoiled-runtimelocally, use the following steps:# Have a local copy of the `coiled-runtime` repositorygitclonehttps://github.com/coiled/coiled-runtimecdcoiled-runtime# Make sure conda-build is installedcondainstall-cconda-forgeconda-build# Build the metapackagecondabuildrecipe-cconda-forge--output-folderdist/conda--no-anaconda-upload# Install the built `coiled-runtime` metapackagecondainstall-c./dist/conda/-cconda-forgecoiled-runtimeTestThecoiled-runtimetest suite can be run locally with the following steps:Ensure your local machine is authenticated to use thedask-engineeringCoiled account and
the Coiled Dask Engineering AWS S3 account.Create a Python environment and install development dependencies as
specified inci/environment.yml.Install a coiled runtime environment. This might be from one of the environments
listed inenvironments/, or it could be a development environment if you are
testing feature branches of dask or distributed. This test suite is configured
to run Coiled'spackage_syncfeature, so your local environment should be copied
to the cluster.Run tests withpython -m pytest testsAdditionally, tests are automatically run on pull requests to this repository.
See the section below on creating pull requests.BenchmarkingThecoiled-runtimetest suite contains a series of pytest fixtures which enable
benchmarking metrics to be collected and stored for historical and regression analysis.
By default, these metrics are not collected and stored, but they can be enabled
by including the--benchmarkflag in your pytest invocation.From a high level, here is how the benchmarking works:Data from individual test runs are collected and stored in a local sqlite database.
The schema for this database is stored inbenchmark_schema.pyThe local sqlite databases are appended to a global historical record, stored in S3.The historical data can be analyzed using any of a number of tools.dashboard.pycreates a set of static HTML documents showing historical data for the tests.Running the benchmarks locallyYou can collect benchmarking data by running pytest with the--benchmarkflag.
This will create a localbenchmark.dbsqlite file in the root of the repository.
If you run a test suite multiple times with benchmarking,
the data will be appended to the database.You can compare with historical data by downloading the global database from S3 first:awss3cps3://coiled-runtime-ci/benchmarks/benchmark.db./benchmark.db
pytest--benchmarkChanging the benchmark schemaYou can add, remove, or modify columns by editing the SQLAlchemy schema inbenchmark_schema.py.
However, if you have a database of historical data, then the schemas of the new and old data will not match.
In order to account for this, you must provide a migration for the data and commit it to the repository.
We usealembicto manage SQLAlchemy migrations.
In the simple case of simply adding or removing a column to the schema, you can do the following:# First, edit the `benchmark_schema.py` and commit the changes.alembicrevision--autogenerate-m"Description of migration"gitaddalembic/versions/name_of_new_migration.py
gitcommit-m"Added a new migration"Migrations are automatically applied in the pytest runs, so you needn't run them yourself.Deleting old test dataAt times you might change a specific test that makes older benchmarking data irrelevant.
In that case, you can discard old benchmarking data for that test by applying a data migration
removing that data:alembicrevision-m"Declare bankruptcy for <test-name>"# Edit the migration here to do what you want.gitaddalembic/versions/name_of_new_migration.py
gitcommit-m"Bankruptcy migration for <test-name>"An example of a migration that does this ishere.Using the benchmark fixturesWe have a number of pytest fixtures defined which can be used to automatically track certain metrics in the benchmark database.
The most relevant ones are summarized here:benchmark_time: Record wall clock time duration.benchmark_memory: Record memory usage.benchmark_task_durations: Record time spent computing, transferring data, spilling to disk, and deserializing data.auto_benchmark_time: Include this fixture to measure the wall clock time of the whole test automatically.benchmark_all: Record all available metrics.For more information on all available fixtures and examples on how to use them, please refer to their documentation.Writing a new benchmark fixture would generally look like:Requesting thetest_run_benchmarkfixture, which yields an ORM object.Doing whatever setup work you need.yielding to the testCollecting whatever information you need after the test is done.Setting the appropriate attributes on the ORM object.Thebenchmark_timefixture provides a fairly simple example.Investigating performance regressionsIt is not always obvious what the cause of a seeming performance regression is.
It could be due to a new version of a direct or transitive dependency,
and it could be due to a change in the Coiled platform.
But often it is due to a change indaskordistributed.
If you suspect that is the case, Coiled'spackage_syncfeature combines well with the benchmarking infrastructure here andgit bisect.The following is an example workflow which could be used to identify a specific commit indaskwhich introduced a performance regression.
This workflow presumes you have two terminal windows open, one with thecoiled-runtimetest suite,
and one with adaskrepository with which to drive bisecting.Create your software environmentYou should create a software environment which can run this test suite, but with an editable install ofdask.
You can do this in any of a number of ways, but one approach coule becondaenvcreate-ntest-env--fileci/environment.yml# Create a test environmentcondaactivatetest-env# Activate your test environmentpipinstall.# Install the `coiled-runtime` metapackage dependencies.(cd<your-dask-dir>&&pipinstall-e.)# Do an editable install for daskStart bisectingLet's say the currentHEADofdaskis known to be bad, and$REFis known to be good. If you are looking at an upstream run where you have access to the static page, you can check the dates reported for each run and do agit logwith the corresponding dates to get a list of commits to use in the bisecting process.git log --since='2022-08-15 14:15' --until='2022-08-18 14:15' --pretty=onelineIn the terminal opened to your dask repository you can initialize a bisect workflow withcd<your-dask-dir>
gitbisectstart
gitbisectbad
gitbisectgood$REFTest for regressionsNow that your editable install is bisecting, run a test or subset of tests which demonstrate the regression in yourcoiled-runtimeterminal.
Presume thattests/benchmarks/test_parquet.py::test_write_wide_datais such a test:pytesttests/benchmarks/test_parquet.py::test_write_wide_data--benchmarkOnce the test is done, it will have written a new entry to a local sqlite filebenchmark.db.
You will want to check whether that entry displays the regression.
Exactly what that check will look like will depend on the test and the regression.
You might have a script that builds a chart frombenchmark.dbsimilar todashboard.py,
or a script that performs some kind of statistical analysis.
But let's assume a simpler case where you can recognize it from theaverage_memory.
You can query that withsqlite3benchmark.db"select dask_version, average_memory from test_run where name = 'test_write_wide_data';"If the last entry displays the regression, mark the commit in your dask terminal asbad:gitbisectbadIf the last entry doesn't display the regression, mark the commit in your dask terminal asgood:gitbisectgoodProceed with this process until you have narrowed it down to a single commit.
Congratulations! You've identified the source of your regression.A/B testingIt's possible to run the Coiled Runtime benchmarks for A/B comparisons.Read full documentation.ContributeThis repository uses GitHub Actions secrets for managing authentication tokens used
to access resources like Coiled clusters, S3 buckets, etc. However, because GitHub Actionsdoesn't
grant access to secrets for forked repositories,please submit pull requests directly from thecoiled/coiled-runtimerepository,
not a personal fork.ReleaseTo issue a newcoiled-runtimerelease:Locally update thecoiled-runtimeversion and package pinnings specified inrecipe/meta.yaml.When updating package version pinnings (in particulardaskanddistributed)
confirm there are no reported large scale stability issues (e.g. deadlocks) or
performance regressions on thedask/distributedissue trackers or offline
reports.Open a pull request to thecoiled-runtimerepository titled "Release X.Y.Z" with these changes
(whereX.Y.Zis replaced with the actual version for the release).After all CI builds have passed the release pull request can be merged.Add a new git tag for the release by following the steps below on your local machine:# Pull in changes from the Release X.Y.Z PRgitcheckoutmain
gitpulloriginmain# Set release version numberexportRELEASE=X.Y.Z# Create and push release taggittag-a$RELEASE-m"Version$RELEASE"gitpushoriginmain--tagsUpdate thecoiled-runtimepackage on conda-forge by opening a pull request to thecoiled-runtimeconda-forge feedstockwhich updates thecoiled-runtimeversion and package version pinnings.Note that pull requests to conda-forge feedstocks must come from a fork.Reset the build number back to0if it isn't already.For more information on updating conda-forge packages, see theconda-forge docs.LicenseBSD-3