Package: pytest-job-selection
pytest-job-selectionByArvid Jakobsson.pytest-job-selectionis apytestplugin for load
balancing test suites.In short, the plugin provides a new pytest argument--jobsuch that
runningpytest --job X/Y [tests...](where1 <= X <= Y) groups the
selectedtestsinYjobs and then executes the tests of jobX. By default, the plugin attempts to create jobs where the number
of test cases are approximately close to each other.To instead obtain jobs whose run time are close to each other, a JUnit
XML file can be provided through the--prev-junit-xml junit.xmlargument. In this mode, the plugin will balance jobs based on the
previously reported run time of test cases injunit.xml. Test cases
lacking an entry injunit.xmlare given a default run time, and
cases injunit.xmlthat are not selected are ignored. SeeJUnit XML
filesfor more information on this feature.This plugin enables convenient parallel execution of large pytest
suites in CIs such asGitLab CIusingparallel
jobs.
See the sectionGitLab CI integrationfor
more details.This plugin is inspired by a similar functionality inTezt, an OCaml test framework.FeaturesAdds a--jobpytest argument that executes a subset of selected
test.Adds a--prev-junit-xml <junit.xml>pytest argument. When
supplied, the plugin heuristically attempts to balance test jobs
using a greedy knapsack algorithm into jobs of even runtime, using
the timing information injunit.xml. This argument can only be
given once, but JUnit XML files can trivially be merged.Adds a--jobs-dry-runpytest argument that outputs debug
information on test balancing.Requirementspytesttyping_extensionsIn addition,cramis used to test the
plugin.InstallationYou can installpytest-job-selectionviapipfromPyPI:$ pip install pytest-job-selectionUsageThe following command should suffice for most use cases:pytest --prev-junit-xml junit.xml --job X/Y [tests...]This will group selected tests intoYjobs. It uses abalancing heuristicto group selected
tests, based on the previously recorded timing information in the
JUnit XML filejunit.xml. It then executes all tests in jobX. Jobs
are 1-indexed, so that job1is the first job and jobYis the last
one. In other words, executing:pytest --prev-junit-xml junit.xml --job 1/Y [tests...]
pytest --prev-junit-xml junit.xml --job 2/Y [tests...]
...
pytest --prev-junit-xml junit.xml --job Y/Y [tests...]Will execute the same tests as:pytest [tests...]The plugin can be used without passing--prev-junit-xml. In this
case, jobs are instead balanced by number of tests.JUnit XML filesSee thepytest
documentationfor more information on JUnit XML files.
In short, an JUnit XML file can be obtained file by executing a test suite with the--junitxml=junit.xmlargument:pytest --junitxml=junit.xml [tests...]If ajunit.xmlis not provided to thepytest-job-selectionplugin
using--prev-junit-xml, then its batching heuristic will assume that
all test cases have the same running time, and consequently attempts
to create batches with a balanced number of test cases.If a test case is not present in the providedjunit.xml, for
instance if a new test case has been added sincejunit.xmlwas
generated, then a default run time of 1 minute will be
assumed. Conversely, test cases presentjunit.xmlwhich are not
selected for execution, for instance test cases that has been removed
sincejunit.xmlwas generated, have no impact on balancing.Consequently, thejunit.xmldoes not have to be up-to-date at each
run, but as a test suite grows, it will become gradually less
well-balanced. It is therefore good practice to update the file at
regular intervals.Dry runYou can simulate balancing run with the--jobs-dry-run --job X/Yflag. This will collect and group tests, and then output:The list of jobs with:weight: its expected run time as per the previously
recorded timing information#classes: the number of test classes/modules in this jobThe full list of test classes/modules sorted by job appartenance
and the weight of that test class/module (its weight is sum of the
running time of all test classes in that class as per the
previously recorded timing information)Job statistics, including:the total number of jobs;the minimum, maximum and average job weight;the minimum, maximum and average number of classes/modules per job.The test classes/modules of the currently selected jobs and their weight.A list oforphanedtest classes. These are test classes that
appear in the JUnit XML file supplied with--prev-junit-xmlbut
which does not correspond to any selected test. The presence of
orphans indicates that the JUnit XML file may be out-of-date, but
it will not impact balancing.A Worked ExampleWe will use a simple dummy test (availablehere:fromtimeimportsleepclassTestExampleA:deftest_a(self):sleep(1)classTestExampleB:deftest_b(self):sleep(2)classTestExampleC:deftest_c(self):sleep(1)classTestExampleD:deftest_d(self):sleep(2)This module contains four test classes that do nothing but sleep for a
given period of time. We can first try run these tests in two jobs
without giving the job selection plugin any previous timings on which
to base balancing:$ pytest --job 1/2 example/example_test.py -v
example/example_test.py::TestExampleA::test_a PASSED [ 50%]
example/example_test.py::TestExampleC::test_c PASSED [100%]
$ pytest --job 2/2 example/example_test.py -v
example/example_test.py::TestExampleB::test_b PASSED [ 50%]
example/example_test.py::TestExampleD::test_d PASSED [100%]This groupsTestExampleAwithTestExampleCandTestExampleBwithTestExampleD. Each job contains an even number of test classes, but
the jobs are unbalanced in terms of run time, as the first job will
run in ~2 seconds while the second will run in ~4 seconds.Note that you can also preview the balancing using the pytest flag--collect-only(here in addition to the--quietflag for terse output):$ pytest --job 1/2 example/example_test.py --collect-only --quiet
(job selection: 1/2 with 4 timings from junit.xml)
example/example_test.py::TestExampleA::test_a
example/example_test.py::TestExampleB::test_b

4 tests collected in 0.01s
$ pytest --job 2/2 example/example_test.py --collect-only --quiet
(job selection: 2/2 with 4 timings from junit.xml)
example/example_test.py::TestExampleC::test_c
example/example_test.py::TestExampleD::test_d

4 tests collected in 0.01sTo balance the jobs based on the expected runtime of individual test
cases, we record ajunit.xmlfile:$ pytest --junitxml=junit.xml example/example_test.pyWe can inspectjunit.xmland verify that it contains the expected
timings:<?xml version="1.0" encoding="utf-8"?><testsuites><testsuitename="pytest"errors="0"failures="0"skipped="0"tests="4"time="6.040"><testcaseclassname="example.example_test.TestExampleA"name="test_a"time="1.002"/><testcaseclassname="example.example_test.TestExampleB"name="test_b"time="2.002"/><testcaseclassname="example.example_test.TestExampleC"name="test_c"time="1.003"/><testcaseclassname="example.example_test.TestExampleD"name="test_d"time="2.004"/></testsuite></testsuites>And then feed the recording into the plugin:$ pytest --prev-junit-xml junit.xml --job 1/2 example/example_test.py -v
example/example_test.py::TestExampleA::test_a PASSED [ 50%]
example/example_test.py::TestExampleD::test_d PASSED [100%]
$ pytest --prev-junit-xml junit.xml --job 2/2 example/example_test.py -v
example/example_test.py::TestExampleB::test_b PASSED [ 50%]
example/example_test.py::TestExampleC::test_c PASSED [100%]This time,TestExampleAis grouped withTestExampleDandTestExampleBis grouped withTestExampleC, giving both jobs an
expected runtime of ~3 seconds each.Finally, we can do a dry run with `--jobs-dry-run`` to obtain an
overview of the obtained balancing:$ pytest --prev-junit-xml junit.xml --job 2/2 --jobs-dry-run example/example_test.py
collecting ... (job selection: 2/2 with 4 timings from junit.xml)
dry run
Jobs: weight and contents
job       weight   #classes
1        0:00:03          2
2        0:00:03          2

Jobs: weight and full contents
job                                  class    weight
1        example.example_test.TestExampleD   0:00:02
1        example.example_test.TestExampleA   0:00:01
2        example.example_test.TestExampleB   0:00:02
2        example.example_test.TestExampleC   0:00:01

Jobs: statistics
jobs_total        weight: avg       min       max   #classes: avg   min   max
jobs_total=2          0:00:03   0:00:03   0:00:03             2.0     2     2
Can add 0:00:00.001000 without increasing wall-time.

Slowest classes (top 10):
weight                                   class
0:00:02      example.example_test.TestExampleD
0:00:02      example.example_test.TestExampleB
0:00:01      example.example_test.TestExampleC
0:00:01      example.example_test.TestExampleA

Would run test classes in job 2/2:
class                                   weight
example.example_test.TestExampleB      0:00:02
example.example_test.TestExampleC      0:00:01Balancing HeuristicThe balancing heuristic is based on a greedy solution to the knapsack
problem. Each test that is missing a previously recorded run time
provided through--prev-junit-xmlwill be assigned a default runtime
of 1 minute. If no previously recorded run times are provided, then
this applies to all test cases, and consequently the heuristic will
balance jobs based only on the number of assigned test cases (i.e.,
attempting to create jobs whose number of test cases are close to
each other).GitLab CI integrationThis plugin can conveniently be used inGitLab
CIto exploitparallel
jobs.For instance, to parallelize a job like:pytest:script:-pytest tests/Install the plugin in the CI, make sure that ajunit.xmlis available
in the repository at e.g.tests/junit.xmland change the job to:pytest:parallel:10script:-pytest --prev-junit-xml tests/junit.xml --job ${CI_NODE_INDEX}/${CI_NODE_TOTAL}"--junitxml=reports/report_${CI_NODE_INDEX}_${CI_NODE_TOTAL}.xml"tests/artifacts:paths:-reportswhen:alwaysThis will split the pytest job into 10 parallel jobs. The--junitxmlargument has also been added to the pytest command so that new JUnit
XML recordings are produced in the CI and then stored as artifacts
using theartifactsstanza.This last point helps when rebalancing the suite, which you can now do
by downloading the recordings from the CI, merging them and commiting
them totests/junit.xml. In the next section, we describe the
included scriptglci_jobs_fetch_reportsthat can be used to
partially automate this process.Retrieving JUnit XML files from GitLab CIUpdating thejunit.xmlfile used for balancing becomes a hassle when
the number of jobs grow. This plugin installs a scriptglci_jobs_fetch_reports(locatedherein this
repository) that can be used to this effect. For usage information,
callglci_jobs_fetch_reports --help.LimitationsTest classesThis plugin balances tests at the granularity of modules or test
classes. All test cases of the same class will always execute in the
same job. Similarly, all test cases in a module that do not correspond
to class methods will execute in the same job.Missing timingsIf timings are missing for test case, then the balancer will silently
assume its running time is 1 minute.Empty jobsIf there are more jobs than test classes or modules to balance, then at
least one job will be empty. In this case,pytestwill
exit with a non-zero error code.ContributingContributions are very welcome. Tests can be run withtox, please ensure the coverage
at least stays the same before you submit a pull request.LicenseDistributed under the terms of theMITlicense,
"pytest-job-selection" is free and open source softwareIssuesIf you encounter any problems, pleasefile an
issuealong with
a detailed description.