Package: casabench
casabenchTracking the performance of CASAThe scope of this repository is the automatic execution of performance benchmarking tests that satisfy two use-cases:To characterize trends in performance metrics acrossCASA6repository history (after releases are published from the main trunk)To detect code causing performance issues during verification of development branches (before changes are merged with the main trunk)To satisfy these use cases, this prototype uses a wrapper script to coordinate calls to theairspeed-velocity(asv) framework, which manages the execution of tests (located in thebenchmarksdirectory). The successful execution of benchmarks produces environment-specific performance data (stored inJSONformat in theresultsdirectory), and further calls toasvgenerate anHTMLrepresentation of the results and automatic regression analysis (published to the web from a dedicated branch viaGitHub pages). This repository could be modified to work with other tools in the future, includingasv's integration with the Python standard libraryprofiler.Right now casabench is pre-alpha software, not intended for general use.InstallationThis package is not yet installable, either from a source or wheel distribution. Early-stage usage relies on creation of a python environment, using eithercondaorvirtualenv; installation ofasv>=0.4.2and dependencies; cloning of the repository onto the target host; and manual calls toasvfrom the command line. This works until the wrapper script exists.The package will be made installable through the specification ofPEP-518-compliant build requirements in the form of setup.py, setup.cfg, and pyproject.toml files. Wheels will be available for download viapypi(and eventually,conda).Setupcasabench installs CASA intoasvruntime environments from wheels published in an NRAO-managed package repository, so the user'spip configurationneeds to have these URLs included in the list that will be searched by the installer. Using the global level ensures that the configuration is respected by test executions inside each of the virtual environments created by asv.[global]
extra-index-url = 
		https://casa-pip.nrao.edu:443/repository/pypi-group/simple
		https://casa-pip.nrao.edu/repository/pypi-casa-release/simpleAn additonal step is required to configure access to test data:git clone --no-checkout https://open-bitbucket.nrao.edu/scm/casa/casatestdata.git
cd casatestdata/
git config core.sparseCheckout true
echo performance/* >> .git/info/sparse-checkout
git checkout masterNote that is not working for all the tests right now since some prototype functions reference data which have not yet been added to the casatestdata repository. This will be fixed in the future. In the meantime, if you know where copies of the files live, you can place them in the expected directory structure:casatestdata/
└── performance
    ├── calibration_runtime
    │   ├── uid___A002_Xe1f219_X6d0b.ms
    │   ├── uid___A002_Xe1f219_X6d0b.ms.hifa_bandpassflag.s12_1.spw16_18_20_22.channel.solintinf.bcal.tbl
    │   ├── uid___A002_Xe1f219_X6d0b.ms.hifa_bandpassflag.s12_3.spw16_18_20_22.solintinf.gacal.tbl
    │   ├── uid___A002_Xe1f219_X6d0b.ms.hifa_bandpassflag.s12_4.spw16_18_20_22.solintint.gpcal.tbl
    │   ├── uid___A002_Xe1f219_X6d0b.ms.h_tsyscal.s6_1.tsyscal.tbl
    │   └── uid___A002_Xe1f219_X6d0b.ms.s12.4.callibrary
    ├── flagdata_runtime
    │   ├── uid___A002_Xe1f219_X6d0b_data_autocorr_WRAY_scan7.ms
    │   └── uid___A002_Xe1f219_X6d0b.flagcmds.txt
    └── tclean_mem_setweighting
        └── uid___A002_Xb9dfa4_X4724_target_spw16.msRunning testsThe first time tests are run on a new machine you will have the option to contribute identifyingmachine informationif the host is not recognized, otherwise sensible defaults will be assigned automatically.Invocation of the wrapper script that callsasvis yet to be completely defined, but will be supplied from the command line interface and have the form:python-mcasabench--use-case1--versionsrevision.txt--push-resultsrepo
python-mcasabench--use-case2--versionsbranches.txt--push-resultswebThese practically represent chained calls toasv, i.e.,asv run --hashfile=revisions.txt
asv publish
git add results && git commit -m "Publish new results" && git pushandasv compare
asv gh-pagesrespectively, but with the complication introduced by our build process properly accounted for by the wrapper script.Saving outputOnce test results have been generated, they can saved by those with repository write access. One usage mode formats and pushes the results files themselves, and another generates static HTML content and either pushes it to the mainwebsiteor launches a local web server for transient analysis of the results.For now, updating results in the repository will only be supported for those with repository write access. Contribution of results via pull request and command line interface will be considered in the future.Alternative publication methods to explore include parsing the JSON ourselves to include it at the bottom of thecasadocs changelog.DetailsWhat the wrapper script doesSince the parts of casa6 this repo tests are split between two sub-repositories and it's complicated to make that (version-specific) build process conform to the project structure thatasvexpects,casatasks(and itscasatoolsdependency) are installed viaairspeed-velocity's dependency handler. This is intended to be refactored once per-commit build and installation of both casatools and casatasks is working properly.Until then, it is necessary to edit theasv.config.jsonconfig file so that the value of the "include" parameter is unary (i.e., we can only test against one version at a time without producing redundant results at a large performance penalty). Then, theasv runcommand can be appropriately invoked, with input to the revision argument that matches the wheel version specified in the config file.For example, to haveasvbuild against CASA version 6.4.0.2, the configuration file must specify"include":[{"python":"3.6", "pip+casatools": "6.4.0.2", "pip+casatasks":"6.4.0.2"}], then all tests can be invoked using:asv run tags/6.4.0.2^! --machine "NRAO workstation"To run a particular class of benchmarks, the--benchparameter is specified, which accepts multiple arguments each of which is handled as a regular expression:asv run --bench "flagdata" -b "calib" tags/6.4.0.2^! --machine "NRAO workstation"The results database (a collection of JSON files) is tracked in the repository. Results files are organized by machine, so tests run from a fresh clone of the repository will build their own database. The HTML is generated from the files stored in the results database. In order to forceasvto write HTML that treats the results as contiguous tests of the same package instead of separate entries in a dependency matrix, it is necessary to remove the build number from the casatools/casatasks specifications in theparamsandrequirementsdictionaries contained in the file corresponding to a given test configuration. For example,...
"params": {"arch": "x86_64", "cpu": "Intel(R) Xeon(R) CPU E5-1660 v4 @ 3.20GHz", "machine": "NRAO workstation", "num_cpu": "16", "os": "Linux 3.10.0-1160.15.2.el7.x86_64", "ram": "32GB", "python": "3.6", "pip+casatools": "6.4.0.2", "pip+casatasks": "6.4.0.2"}, 
"requirements": {"pip+casatools": "6.4.0.2", "pip+casatasks": "6.4.0.2"},
...is changed to...
"params": {"arch": "x86_64", "cpu": "Intel(R) Xeon(R) CPU E5-1660 v4 @ 3.20GHz", "machine": "NRAO workstation", "num_cpu": "16", "os": "Linux 3.10.0-1160.15.2.el7.x86_64", "ram": "32GB", "python": "3.6", "pip+casatools": "6.4.0", "pip+casatasks": "6.4.0"}, 
"requirements": {"pip+casatools": "6.4.0", "pip+casatasks": "6.4.0"},
...Someday in the future it would be great to refactor casabench to build bothcasatoolsandcasatasksfrom source, since that's what the framework is designed to do and it would make the wrapper script redundant. In practice, after many attempts, thecasatoolsbuild process proved too cumbersome to integrate withasvin this fashion (we gotcasatasksto work, but since there is a requirement that both tools and tasks are at the same version, it was equally complicated to manage so we decided to follow the current architecture).