Package: pytest-check
pytest-checkA pytest plugin that allows multiple failures per test.Normally, a test function will fail and stop running with the first failedassert.
That's totally fine for tons of kinds of software tests.
However, there are times where you'd like to check more than one thing, and you'd really like to know the results of each check, even if one of them fails.pytest-checkallows multiple failed "checks" per test function, so you can see the whole picture of what's going wrong.InstallationFrom PyPI:$ pip install pytest-checkFrom conda (conda-forge):$ conda install -c conda-forge pytest-checkExampleQuick example of where you might want multiple checks:importhttpxfrompytest_checkimportcheckdeftest_httpx_get():r=httpx.get('https://www.example.org/')# bail if bad status codeassertr.status_code==200# but if we get to here# then check everything else without stoppingwithcheck:assertr.is_redirectisFalsewithcheck:assertr.encoding=='utf-8'withcheck:assert'Example Domain'inr.textImport vs fixtureThe example above used import:from pytest_check import check.You can also grabcheckas a fixture with no import:deftest_httpx_get(check):r=httpx.get('https://www.example.org/')...withcheck:assertr.is_redirect==False...Validation functionscheckalso helper functions for common checks.
These methods do NOT need to be inside of awith check:block.FunctionMeaningNotesequal(a, b, msg="")a == bnot_equal(a, b, msg="")a != bis_(a, b, msg="")a is bis_not(a, b, msg="")a is not bis_true(x, msg="")bool(x) is Trueis_false(x, msg="")bool(x) is Falseis_none(x, msg="")x is Noneis_not_none(x, msg="")x is not Noneis_in(a, b, msg="")a in bis_not_in(a, b, msg="")a not in bis_instance(a, b, msg="")isinstance(a, b)is_not_instance(a, b, msg="")not isinstance(a, b)almost_equal(a, b, rel=None, abs=None, msg="")a == pytest.approx(b, rel, abs)pytest.approxnot_almost_equal(a, b, rel=None, abs=None, msg="")a != pytest.approx(b, rel, abs)pytest.approxgreater(a, b, msg="")a > bgreater_equal(a, b, msg="")a >= bless(a, b, msg="")a < bless_equal(a, b, msg="")a <= bbetween(b, a, c, msg="", ge=False, le=False)a < b < cbetween_equal(b, a, c, msg="")a <= b <= csame asbetween(b, a, c, msg, ge=True, le=True)raises(expected_exception, *args, **kwargs)Raises given exceptionsimilar topytest.raisesfail(msg)Log a failureNote: This is a list of relatively common logic operators. I'm reluctant to add to the list too much, as it's easy to add your own.The httpx example can be rewritten with helper functions:deftest_httpx_get_with_helpers():r=httpx.get('https://www.example.org/')assertr.status_code==200check.is_false(r.is_redirect)check.equal(r.encoding,'utf-8')check.is_in('Example Domain',r.text)Which you use is personal preference.Defining your own check functionsUsing@check.check_funcThe@check.check_funcdecorator allows you to wrap any test helper that has an assert statement in it to be a non-blocking assert function.frompytest_checkimportcheck@check.check_funcdefis_four(a):asserta==4deftest_all_four():is_four(1)is_four(2)is_four(3)is_four(4)Usingcheck.fail()Using@check.check_funcis probably the easiest.
However, it does have a bit of overhead in the passing cases
that can affect large loops of checks.If you need a bit of a speedup, use the following style with the help ofcheck.fail().frompytest_checkimportcheckdefis_four(a):__tracebackhide__=Trueifa==4:returnTrueelse:check.fail(f"check{a}== 4")returnFalsedeftest_all_four():is_four(1)is_four(2)is_four(3)is_four(4)Using raises as a context managerraisesis used as context manager, much likepytest.raises. The main difference being that a failure to raise the right exception won't stop the execution of the test method.frompytest_checkimportcheckdeftest_raises():withcheck.raises(AssertionError):x=3assert1<x<4Pseudo-tracebacksWithcheck, tests can have multiple failures per test.
This would possibly make for extensive output if we include the full traceback for
every failure.
To make the output a little more concise,pytest-checkimplements a shorter version, which we call pseudo-tracebacks.For example, take this test:deftest_example():a=1b=2c=[2,4,6]check.greater(a,b)check.less_equal(b,a)check.is_in(a,c,"Is 1 in the list")check.is_not_in(b,c,"make sure 2 isn't in list")This will result in:=================================== FAILURES ===================================
_________________________________ test_example _________________________________
FAILURE:
assert 1 > 2
  test_check.py, line 14, in test_example() -> check.greater(a, b)
FAILURE:
assert 2 <= 1
  test_check.py, line 15, in test_example() -> check.less_equal(b, a)
FAILURE: Is 1 in the list
assert 1 in [2, 4, 6]
  test_check.py, line 16, in test_example() -> check.is_in(a, c, "Is 1 in the list")
FAILURE: make sure 2 isn't in list
assert 2 not in [2, 4, 6]
  test_check.py, line 17, in test_example() -> check.is_not_in(b, c, "make sure 2 isn't in list")
------------------------------------------------------------
Failed Checks: 4
=========================== 1 failed in 0.11 seconds ===========================Red outputThe failures will also be red, unless you turn that off with pytests--color=no.No outputYou can turn off the failure reports with pytests--tb=no.Stop on Fail (maxfail behavior)Setting-xor--maxfail=1will cause this plugin to abort testing after the first failed check.Setting-maxfail=2or greater will turn off any handling of maxfail within this plugin and the behavior is controlled by pytest.In other words, themaxfailcount is counting tests, not checks.
The exception is the case of1, where we want to stop on the very first failed check.any_failures()Useany_failures()to see if there are any failures.One use case is to make a block of checks conditional on not failing in a previous set of checks:frompytest_checkimportcheckdeftest_with_groups_of_checks():# always check thesecheck.equal(1,1)check.equal(2,3)ifnotcheck.any_failures():# only check these if the above passedcheck.equal(1,2)check.equal(2,2)SpeedupsIf you have lots of check failures, your tests may not run as fast as you want.
There are a few ways to speed things up.--check-max-tb=5- Only first 5 failures per test will include pseudo-tracebacks (rest without them).The example shows5but any number can be used.pytest-check uses custom traceback code I'm calling a pseudo-traceback.This is visually shorter than normal assert tracebacks.Internally, it uses introspection, which can be slow.Allowing a limited number of pseudo-tracebacks speeds things up quite a bit.Default is 1.Set a large number, e.g: 1000, if you want pseudo-tracebacks for all failures--check-max-report=10- limit reported failures per test.The example shows10but any number can be used.The test will still have the total nuber of failures reported.Default is no maximum.--check-max-fail=20- Stop the test after this many check failures.This is useful if your code under test is slow-ish and you want to bail early.Default is no maximum.Any of these can be used on their own, or combined.Recommendation:Leave the default, equivelant to--check-max-tb=1.If excessive output is annoying, set--check-max-report=10or some tolerable number.Local speedupsThe flags above are global settings, and apply to every test in the test run.Locally, you can set these values per test.Fromexamples/test_example_speedup_funcs.py:deftest_max_tb():check.set_max_tb(2)foriinrange(1,11):check.equal(i,100)deftest_max_report():check.set_max_report(5)foriinrange(1,11):check.equal(i,100)deftest_max_fail():check.set_max_fail(5)foriinrange(1,11):check.equal(i,100)ContributingContributions are very welcome. Tests can be run withtox.
Test coverage is now 100%. Please make sure to keep it at 100%.
If you have an awesome pull request and need help with getting coverage back up, let me know.LicenseDistributed under the terms of theMITlicense, "pytest-check" is free and open source softwareIssuesIf you encounter any problems, pleasefile an issuealong with a detailed description.ChangelogSeechangelog.md