Package: partialtesting
Partial TestingContinuous Integration practices enable the frequent and verified (built and tested) contribution of code to shared repositories. But, as projects grow, so do their test suites and the time it takes to run them.
Small and contained changes do not require all tests to be run, most of them would not be relevant. To improve developer productivy and reduce resource usage, we have implementedPartial Testing.
By recording coverage data from previous runs, it identifies which tests should be run for each new change or pull-request.To maintain coverage data up-to-date, and to ensure all tests are still occasionally run, master branch builds execute the entire test suite while recording coverage at the same time. The coverage data from the master branch is saved as a.coveragefile (which is a sqlite DB file fromcoveragepy>=5.0, details availablehere) and non-master branches utilize that file to run only the relevant tests for the changes included in the branch.For simplicity, we consider a file as the smallest possible changed unit (instead of doing it at the line level). That is, if a line has changed, it is treated as if the entire file has changed.
This approach reduces the complexity of Partial Testing and provides a wider safety net against filtering out tests that were actually relevant.As shown on the image above, once a file has been modified, a few different testing scenarios arise according to the type of the file. It is also possible that a combination is required.
The different file types are as follows:Partial Testing only works with projects that are ingit.Source Code FileIncludes any file that contains code and is not part of the test suite. They usually have the Python extension,.py, but could also have.cpp,.c, etc. instead.When modified:a) check the saved coverage data to find all relevant tests (test names).b) For each test name, find which file has its definition.c) feedpytest(or any other testing tool) the files from step b.When a new file is added, the complete test suite must be run.Test Code FileIncludes any file that contains code and is part of the test suite, by default it includes all python files undertests/. They have the Python extension.py. When modified, run all the tests contained within the changed file.When a new test file is added, all the tests within itself must be run. When one is deleted, no tests from it need to be run.Additionally, some files under the tests directory might be utility files that get imported by other test files. Because of this, for modified test files, we need 2 steps:a) check which tests use the file (just like non-test code files)b) run the tests within the modified fileSpecial/Config FileIncludes any file that does not contain code but affects the way code is run. Examples:setup.cfgWhen modified or when a new one is added, all tests need to be run.Each project should have the ability to specify which files and/or file extensions should be considered special files, see below for details.We have includedconftest.py(pytest) under this category of files, even though it initially looks like a code file (.py) or a test file (it sits under the test directories).Non-Code FileIncludes any file that does not contain code. For example, README or.mdfiles.When modified or a new one is added, no tests need to be run.In Summary,Source Code FilesTest FilesNon-Code FilesSpecial/Config FilesExtension.py, .cpp, .c.py.md.cfgModifiedRun tests that use the fileRun tests a) within modified file b) that use the fileDon't run testsRun all testsAdded NewRun all testsRun tests within new fileDon't run testsRun all testsDeletedRun tests that use the fileDo not run that fileDon't run testsRun all testsUsageInstallationpip install partialtesting  # soon to be published in pypiOnce installed, the binariespartialtestandpartialtesting(they are identical, the later is kept for compatibility) become available in your virtual environment:$ partialtesting --help
Usage: partialtesting [OPTIONS]

  Partial Testing (PT) identifies which tests need to be run for a given
  change set, improving the speed of testing, developer productivity and
  resource usage. To achieve this, PT relies on enriched coverage data
  generated before hand, generally by a master-branch build running on a CI
  (e.g. Jenkins).

  PT expects to find the coverage data in a directory with the below
  pattern: <coverage_dir>/<project_name>. Use the options for this script to
  specify them

  More information available in the README.

Options:
  --coverage-dir TEXT        Path to the saved coverage data. Set a default
                             path by setting the below in ~/.partialtesting:
                             [coverage] dir=<path>

  --project-name TEXT        Project name (e.g. numpy).The name will be used
                             to get the path to the coverage data for this
                             project:
                             <coverage_dir>/<project_name>/.../.coverage
                             [required]Partialtesting LocallyThis sections shows how to usepartialtestinglocally, but its convenience and benefits are maximized when used with a continuous integration system (CI), like Jenkins. That will be explained in the section below.To usepartialtestinglocally, specify the directory in which the coverage data will be stored. That is, the coverage data generated during a previous run of the entire test suite or from a master build, more details in the CI section above.As an example, say your project's code repository (git) is as follows:$ tree
├── code1.py
├── code2.py
├── .coveragerc  # explained below
└── tests
    ├── __init__.py
    └── test_code1.pyand the files have the following content:$ cat code1.py
def myfunc1():
    print('in myfunc1')

def myfunc2():
    print('in myfunc2')
    myfunc3()

def myfunc3():
    print('in myfunc3')$ cat code2.py
def myfunc4():
    print('in myfunc4')$ cat tests/test_code1.py
import code1

def test_code1():
    code1.myfunc1()

def test_code2():
    code1.myfunc2()Now, run the entire test suite to generate the coverage data thatpartialtestingwill use.$ pip install coverage>=5 pytest
$ PYTHONPATH='.' coverage --branch run -m pytest tests/
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/fersarr/workspace/fake_project
collected 2 items

tests/test_code1.py ..                                                                                                                                                             [100%]

=================================================================================== 2 passed in 0.01s ====================================================================================Make sure you havecoverage>=5.0and a.coveragercfile telling coverage to record test contexts (read more here)$ cat .coveragerc
[run]
dynamic_context = test_functionThecoverage ... pytestcommand should have generated a.coveragefile, which is a SQLite database:$ sqlite3 .coverage ".tables"
arc              coverage_schema  line_bits        tracer
context          file             metaNow, move the.coveragefile into a directory that will hold coverage data for this project:$ mkdir -p saved_coverage/fake_project/ && cp .coverage saved_coverage/fake_project/
$ tree
├── saved_coverage
│   └── fake_project
│       └── .coverageOkay, everything is set and ready forpartialtestingto work, so let's modify a file and runPT.Modified a code file by adding one line:$ git diff
diff --git a/code1.py b/code1.py
index faecfed..40f81cc 100644
--- a/code1.py
+++ b/code1.py
@@ -1,5 +1,6 @@
 def myfunc1():
     print('in myfunc1')
+    print('changing myfunc1')Runpartialtesting:$ partialtest --coverage-dir saved_coverage --project-name fake_project --compare-to-branch master
...
INFO:root:Partial Testing: using coverage file 'saved_coverage/fake_project/.coverage'
...
INFO:root:Creating file test_files_to_run.txt
INFO:root:Partial Testing: relevant test files:
tests/test_code1.pyAsPTtells us, for the change we made, we need to run the tests intests/test_code1.py.Partialtesting on a CI (e.g. Jenkins)As mentioned above, partialtesting can be used in a Continous Integration system (CI), like Jenkins to improve resource usage and reduce build time. The process is similar to running it locally as explained in the section above, but the difference lies in setting up the master branch to produce the coverage data.coverageand saving it in an accessible directory so that non-master branches can runpartialtestingand use it.Run the full test suite with coverage in the master branch and save the.coveragefile to a path:$ coverage run -p --branch pytest tests/
$ cp .coverage jenkins/saved_coverage/project_x/907/  # build number 907Reminder: add/create a.coveragercfile as exlpained above to save test contexts with the coverage data.Usepartialtestingon the non-master branch to get a list of the tests that should be run given the changes in the branch.partialtesting --project-name project_x --coverage-dir /jenkins/saved_coverage/ --git-diff-use-head --special-files "'setup.py', 'setup.cfg'" --special-extensions "'.pkl', '.h5', '.csv'"The list of tests that need to be run is output to the filetest_files_to_run.txt(customisable via--output-file)Feed those tests topytestor your preferred testing tool.AcknowledgementsPartial Testing has been under active development atMan Alpha Techsince 2019.Original concept and implementation:Fernando Saravia RajalContributors:Fernando Saravia RajalLuke FitzgeraldRelated Linkswho-tests-what in coverage.pywho-tests-what is here coverage v5who-tests-what empty context namesslower tests when recording coverage contextpytest-testmonpytest-testmon pycon videowho-tests-what issue with old-style classes in PY2