Professionelles Testen für Python mit pytest
Florian Bruhin / “The Compiler”

15. April 2021
1

Before we get started...
About you!

I like writing tests

I regularly use pytest

I usually don’t write tests

I regularly use unittest.py

2

Before we get started...
About meLike tests

Don’t write tests

pytestunittest.py

2011 Started using Python
2013 Started developing qutebrowser, writing tests
2015 Switched to pytest, ended up as a maintainer
40% employed (OST: Eastern Switzerland University of Applied Sciences),
60% open-source and freelancing (Bruhin Software)3

Before we get started...
About meLike tests

Don’t write tests

pytestunittest.py

2011 Started using Python
2013 Started developing qutebrowser, writing tests
2015 Switched to pytest, ended up as a maintainer
40% employed (OST: Eastern Switzerland University of Applied Sciences),
60% open-source and freelancing (Bruhin Software)3

Before we get started...
About meLike tests

Don’t write tests

pytestunittest.py

2011 Started using Python
2013 Started developing qutebrowser, writing tests
2015 Switched to pytest, ended up as a maintainer
40% employed (OST: Eastern Switzerland University of Applied Sciences),
60% open-source and freelancing (Bruhin Software)3

Before we get started...
About meLike tests

Don’t write tests

pytestunittest.py

2011 Started using Python
2013 Started developing qutebrowser, writing tests
2015 Switched to pytest, ended up as a maintainer
40% employed (OST: Eastern Switzerland University of Applied Sciences),
60% open-source and freelancing (Bruhin Software)3

Before we get started...
About meLike tests

Don’t write tests

pytestunittest.py

2011 Started using Python
2013 Started developing qutebrowser, writing tests
2015 Switched to pytest, ended up as a maintainer
40% employed (OST: Eastern Switzerland University of Applied Sciences),
60% open-source and freelancing (Bruhin Software)3

4

Why pytest?
Features• Automatic test discovery,
no-boilerplate test code(boilerplate: repeated code without any “real” use)

• Useful information when a test fails• Test parametrization• Modular setup/teardown via fixtures• Customizable: Many options, hundreds ofuseful plugins

5

Why pytest?
Popularity• Automatic test discovery,
no-boilerplate test code(boilerplate: repeated code without any “real” use)

• Useful information when a test fails• Test parametrization• Modular setup/teardown via fixtures• Customizable: Many options, hundreds ofuseful plugins(JetBrains Python Developers Survey 2020)5

No boilerplate
Example codedef add_two(val):
return val + 2

6

No boilerplate
Test with unittest.py

import unittest
class AddTwoTests(unittest.TestCase):
def testAddTwo(self):
self.assertEqual(add_two(2), 4)
if __name__ == '__main__':unittest.main()7

No boilerplate
Test with pytest

import unittest
class AddTwoTests(unittest.TestCase):
def testAddTwo(self):
self.assertEqual(add_two(2), 4)

def test_add_two():
assert add_two(2) == 4

if __name__ == '__main__':unittest.main()8

No boilerplate
Assert introspection

assert x
assert x == 1
assert x != 2
assert not x
assert x < 3 or y > 5# with unittest.py:# assertTrue(x)# assertEqual(x, 1)# assertNotEqual(x, 2)# assertFalse(x)# ?

9No boilerplate
Failing tests____________________________ test_failure ____________________________>
EEEE

def test_failure():a = "Hello World!"b = "Hello, World!"
assert a == bAssertionError: assert 'Hello World!' == 'Hello, World!'- Hello, World!
?
+ Hello World!

test_output.py:4: AssertionError====================== short test summary info =======================
FAILED test_output.py::test_failure - AssertionError: assert 'Hello...
========================= 1 failed in 0.05s ==========================
10

No boilerplate
Other output examples>
EE
E

def test_eq_list():assert [0, 1, 2] == [0, 1, 3]
assert [0, 1, 2] == [0, 1, 3]
At index 2 diff: 2 != 3
Use -v to get the full diff11

No boilerplate
Other output examples>
EE
E

def test_eq_list():assert [0, 1, 2] == [0, 1, 3]
assert [0, 1, 2] == [0, 1, 3]
At index 2 diff: 2 != 3
Use -v to get the full diff>EEEE

def test_not_in_text():
text = "single foo line"
assert "foo" not in text
AssertionError: assert 'foo' not in 'single foo line''foo' is contained here:
single foo line?
+++
11Markers

12Markers
Skipping@pytest.mark.skipif(sys.platform != 'win32',
reason="Only runs on Windows",)
def test_windows_features():
assert False13

Markers
Skipping@pytest.mark.skipif(sys.platform != 'win32',
reason="Only runs on Windows",)
def test_windows_features():
assert False======================== test session starts =========================collecting ... collected 1 item
test_skipping.py::test_windows_features SKIPPED (Only runs...) [100%]========================= 1 skipped in 0.00s =========================
13

Markers
Expected to fail@pytest.mark.xfail(reason="See JIRA-2342")
def test_broken_api():assert False14

Markers
Expected to fail@pytest.mark.xfail(reason="See JIRA-2342")
def test_broken_api():assert False======================== test session starts =========================
collecting ... collected 1 item
test_xfail.py::test_broken_api XFAIL (See JIRA-2342)[100%]========================= 1 xfailed in 0.01s =========================

14

Markers
Unexpected pass@pytest.mark.xfail(reason="See JIRA-2342")
def test_broken_api():pass======================== test session starts =========================
collecting ... collected 1 item
test_xpass.py::test_broken_api XPASS (See JIRA-2342)[100%]========================= 1 xpassed in 0.00s =========================

15

Markers
Custom markers@pytest.mark.slow
def test_slow():
time.sleep(2)def test_fast_1():
pass
def test_fast_2():
pass16

Markers
Custom markers@pytest.mark.slow
def test_slow():
time.sleep(2)[pytest]markers =slow: Tests which take a while to run...

def test_fast_1():
pass
def test_fast_2():
pass16

Markers
Custom markers@pytest.mark.slow
def test_slow():
time.sleep(2)[pytest]markers =slow: Tests which take a while to run...

def test_fast_1():
pass$ pytest -m "not slow"=============== test session starts ================
collected 3 items / 1 deselected / 2 selected

def test_fast_2():
pass

test_custom_marker.py::test_fast_1 PASSED
test_custom_marker.py::test_fast_2 PASSED========= 2 passed, 1 deselected in 0.00s =========
16

Markers
Custom markers with data

@pytest.mark.use_config("production.yml")
def test_prod_config():...

17

Parametrizing

18

Parametrizing
with unittest.py

import unittest
class AddTests(unittest.TestCase):
def testMinusOne(self):
self.assertEqual(add_two(-1), 1)
def testZero(self):
self.assertEqual(add_two(0), 2)
def testTwo(self):
self.assertEqual(add_two(2), 4)
19

Parametrizing
with pytest

import unittest
class AddTests(unittest.TestCase):
def testMinusOne(self):
self.assertEqual(add_two(-1), 1)
def testZero(self):
self.assertEqual(add_two(0), 2)
def testTwo(self):
self.assertEqual(add_two(2), 4)

import pytest@pytest.mark.parametrize('inp, out', [(-1, 1),(0, 2),(2, 4),])
def test_add_two(inp, out):
assert add_two(inp) == out

19

Parametrizing
Result@pytest.mark.parametrize('inp, out', [(-1, 1), (0, 2), (2, 4)])
def test_add_two(inp, out):
assert add_two(inp) == out======================== test session starts =========================[...]
test_param.py::test_add_two[-1-1] PASSED
test_param.py::test_add_two[0-2] PASSED
test_param.py::test_add_two[2-4] PASSED[ 33%][ 66%][100%]========================= 3 passed in 0.01s ==========================
20

Fixtures21

Fixtures
Basic example

import pytest@pytest.fixture
def answer():
return 42
def test_answer(answer):
assert answer == 4222

Fixtures
Basic example

import pytest@pytest.fixture
def answer():
return 42
def test_answer( answer ):assert answer == 42

23

Fixtures
Fixtures using fixtures

import pytest@pytest.fixture
def half():
return 21@pytest.fixture
def answer( half ):return half * 2
def test_answer(answer):
assert answer == 4224

Fixtures
Setup and teardown@pytest.fixture
def database():db = Database()db.connect()yield dbdb.rollback()
def test_database(database):...

25

Fixtures
Other features• Caching fixture values: @pytest.fixture(scope="module")• Using fixtures implicitly: @pytest.fixture(autouse=True)• Running tests with differently configured resources:@pytest.fixture(params=[Postgres(), MariaDB()])

26

Builtin fixtures
tmp_path

def test_one(tmp_path):
input_file = tmp_path / "data.txt"
input_file.write_text("Hello World")
def test_two(tmp_path):
assert not list(tmp_path.iterdir())

27

Builtin fixtures
tmp_path

def test_one(tmp_path):
input_file = tmp_path / "data.txt"
input_file.write_text("Hello World")
def test_two(tmp_path):
assert not list(tmp_path.iterdir())/tmp/
pytest-of-florian/pytest-1/
test_one0/
data.txt
test_two0/
pytest-2/
test_one0/
data.txt
test_two0/27

Builtin fixturesmonkeypatch

def test_debug_mode(monkeypatch):monkeypatch.setenv('MYAPP_DEBUG', '1')
assert 'MYAPP_DEBUG' in os.environ
def test_something_else():assert 'MYAPP_DEBUG' not in os.environ

28

Builtin fixturesmonkeypatch

def test_debug_mode(monkeypatch):monkeypatch.setenv('MYAPP_DEBUG', '1')
assert 'MYAPP_DEBUG' in os.environ
def test_something_else():assert 'MYAPP_DEBUG' not in os.environ
def test_fake_windows(monkeypatch):monkeypatch.setattr(sys, 'platform', 'win32')...

28

Builtin fixtures
Capturingdef test_output(capsys):
print("Hello World")
stdout, stderr = capsys.readouterr()
assert stdout == "Hello World\n"29

Plugins

30

Plugins / Related projects
coverage.py / pytest-cov31

Plugins / Related projects
pytest-bdd

Scenario: Publishing the article
Given I'm an author user
And I have an article
When I go to the article pageAnd I press the publish button
Then no error should be shown
And the article should be published32

Plugins / Related projects
pytest-bdd

Scenario: Publishing the article
Given I'm an author user
And I have an article
When I go to the article pageAnd I press the publish button
Then no error should be shown
And the article should be published@when("I go to the article page")def go_to_article(article, browser):browser.visit(article.url())@when("I press the publish button")def publish_article(browser):browser.find_by_id(...).click()32

Plugins / Related projects
Hypothesis@given(text())
def test_decode_inverts_encode(s):assert decode(encode(s)) == s33

Plugins / Related projects
Hypothesis@given(text())
def test_decode_inverts_encode(s):assert decode(encode(s)) == sFalsifying example: test_decode_inverts_encode(s='')UnboundLocalError: local variable 'character' referenced
before assignment

33

Plugins / Related projects
Plugins, plugins, plugins...

• Property-based testing: hypothesis• Customized reporting: pytest-html, pytest-sugar, pytest-instafail, pytest-emoji• Repeating tests: pytest-repeat, pytest-rerunfailures, pytest-benchmark• Framework/Language integration: pytest-twisted, pytest-django, pytest-qt,pytest-asyncio, pytest-cpp• Coverage and mock integration: pytest-cov, pytest-mock• Other: pytest-bdd (behaviour-driven testing), pytest-xdist (distributed testing)• ... >800 more:https://docs.pytest.org/en/latest/reference/plugin_list.html

34

Plugins / Related projects
Plugins are easy!

# conftest.pydef pytest_addoption(parser):parser.addoption("–backend", choices=("webkit", "webengine"),default="webkit")@pytest.fixture
def backend_arg(request):
return request.config.getoption("–backend")# test_something.py
def test_something(backend_arg):
# ...
35

Plugins / Related projects
Plugins are easy!

# conftest.py
def pytest_addoption(parser):parser.addoption("–backend", choices=("webkit", "webengine"),default="webkit")@pytest.fixture
def backend_arg(request):
return request.config.getoption("–backend")# test_something.py
def test_something(backend_arg):
# ...
35

Plugins / Related projects
Plugins are easy!

# conftest.py
def pytest_addoption(parser):parser.addoption("–backend", choices=("webkit", "webengine"),default="webkit")@pytest.fixture
def backend_arg(request):
return request.config.getoption("–backend")# test_something.py
def test_something(backend_arg):
# ...
35

Plugins / Related projects
Domain-specific languages- name: mp3-compression
type: check-compression
codec: mp3
inputfile: data1.wav
compression: 10%36

Plugins / Related projects
Domain-specific languages<!DOCTYPE html><!– target: hello.txt –>- name: mp3-compression
type: check-compression
codec: mp3
inputfile: data1.wav
compression: 10%<html><head>...</head><body><a href="/data/hello.txt" id="link">
Follow me!</a></body></html>36

Where to go from there...

37

Switching to pytest• pytest can run existing unittest.py and nose tests!
• If you want to rewrite your tests, unittest2pytest can help:class TestExample(unittest.TestCase):
def testExample(self):
self.assertEqual(1, 2)→def testExample(self):
assert 1 == 238

Trainings• As part of enterPy• 22nd April (09:00 – 16:00, fully booked)• 20th May (09:00 – 16:00)• In-house at your company• Three-day pytest/tox/devpi deep-dive• Various other topics (Python basics, advanced Python, best practices, GUIapplications with Qt, ...)• Or tailored to your needs!

39

Contact and resources
Florian Bruhin
florian@bruhin.software
https://bruhin.software/@the_compiler on TwitterAre you interested in customized trainings, development or consulting? Let’s talk!
https://pytest.org
https://github.com/pytest-dev/pytest
irc.freenode.net → #pylib

40