Package: slipcover
SlipCover: Near Zero-Overhead Python Code CoveragebyJuan Altmayer PizzornoandEmery Bergerat UMass Amherst'sPLASMA lab.About SlipcoverSlipCover is a fastcode coveragetool.
It tracks a Python program as it runs and reports on the parts that executed and
those that didn't.
That can help guide your testing (showing code that isn't being tested), debugging,fuzzingor to find "dead" code.Past code coverage tools can make programs significantly slower;
it is not uncommon for them to take twice as long to execute.
SlipCover aims to provide the same information withnear-zero overhead, often
almost as fast as running the original Python program.How it worksPrevious coverage tools likeCoverage.pyrely onPython's tracing facilities,
which add significant overhead.
Instead, SlipCover uses just-in-time instrumentation and de-instrumentation.
When SlipCover gathers coverage information, it modifies the program's Python byte codes,
inserting instructions that let it keep track the lines executed by the program.
As the program executes, SlipCover gradually removes instrumentation that
is no longer needed, allowing those parts to run at full speed.
Care is taken throughout SlipCover to keep things as efficient as possible.
On Python 3.12, rather than rewrite bytecode, SlipCover uses the newsys.monitoringAPI
to collect coverage information.PerformanceThe first image on the right shows SlipCover'sspeedup,
ranging from 1.1x to 3.4x, in relation toCoverage.py, running onCPython 3.10.5.The first two benchmarks are the test suites forscikit-learnandFlask;
"sudoku" runsPeter Norvig's Sudoku solverwhile the others were derived from thePython Benchmark Suite.More "Python-intensive" programs such as sudoku and those from the benchmark
suite (with a larger proportion of execution time spent in Python, rather than in native code)
generate more tracing events, causing more overhead in Coverage.py.
While each program's structure can affect SlipCover's ability to de-instrument,
its running time stays relatively close to the original.OnPyPy 3.9, the speedup ranges from 2.1x to 104.9x.
Since it is so high for some of the benchmarks, we plot it on a logarithmic scale (see the second image on the right).In a proof-of-concept integration with a property-based testing package,
SlipCover sped up coverage-based testing 22x.AccuracyWe verified SlipCover's accuracy againstCoverage.pyand against asimple scriptof our own that collects coverage using Python tracing.
We found SlipCover's results to be accurate, in fact, in certain casesmore accurate.Getting startedSlipCover is available fromPyPI.
You can install it like any other Python module withpip3 install slipcoverYou could then run your Python script with:python3 -m slipcover myscript.pyUsing it with a test harnessSlipCover can also execute a Python module, as in:python3 -m slipcover -m pytest -x -vwhich startspytest, passing it any options (-x -vin this example)
after the module name.
No plug-in is required for pytest.Usage example$python3-mslipcover-mpytest================================================================ test session starts ================================================================platform darwin -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0rootdir: /Users/juan/project/wally/d2k-5, configfile: pytest.iniplugins: hypothesis-6.39.3, mock-3.7.0, repeat-0.9.1, doctestplus-0.12.0, arraydiff-0.5.0collected 439 itemstests/box_test.py .........................                                                                                                   [  5%]tests/image_test.py ...............                                                                                                           [  9%]tests/network_equivalence_test.py .........................................s................................................................. [ 33%]..............................................................................                                                                [ 51%]tests/network_test.py ....................................................................................................................... [ 78%]...............................................................................................                                               [100%]=================================================== 438 passed, 1 skipped, 62 warnings in 48.43s ====================================================File                                 #lines    #miss    Cover%  Lines missing---------------------------------  --------  -------  --------  ------------------------d2k/__init__.py                           3        0       100d2k/box.py                              105       27        74  73, 142-181d2k/image.py                             38        4        89  70-73d2k/network.py                          359        1        99  236tests/box_test.py                       178        0       100tests/darknet.py                        132       11        91  146, 179-191tests/image_test.py                      45        0       100tests/network_equivalence_test.py       304       30        90  63, 68, 191-215, 455-465tests/network_test.py                   453        0       100$As can be seen in the coverage report, d2k lacks some coverage, especially in
itsbox.pyandimage.pycomponents.PlatformsOur GitHub workflows run the automated test suite on Linux, MacOS and Windows, but
really it should work anywhere where CPython/PyPy does.ContributingSlipCover is under active development; contributions are welcome!
Please also feel free tocreate a new issuewith any suggestions or issues you may encounter.Technical InformationFor more details about how SlipCover works please see the following paper, published atISSTA'23:SlipCover: Near Zero-Overhead Code Coverage for Python.AcknowledgementsLogo design bySophia Berger.This material is based upon work supported by the National Science
Foundation under Grant No. 1955610. Any opinions, findings, and
conclusions or recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of the National
Science Foundation.