Package: pytest-regtest
HomeAboutpytest-regtestis a plugin forpytestto implementregression testing.Unlikefunctional testing,regression testingtesting does not test whether the software produces the correct
results, but whether it behaves as it did before changes were introduced.More specifically,pytest-regtestprovidessnapshot testing, which
implements regression testing by recording the textual output of a test
function and comparing this recorded output to a reference output.Regression testingis a common technique to implement basic testing
before refactoring legacy code that lacks a test suite.Snapshot testing can also be used to implement tests for complex outcomes, such
as recording textual database dumps or the results of a scientific analysis
routine.InstallationTo install and activate this plugin execute:$ pip install pytest-regtestBasic UsageWrite a testThepytest-regtestplugin provides multiple fixtures.
To record output, use the fixtureregtestthat works like a file handle:deftest_squares(regtest):result=[i*iforiinrange(10)]# one way to record output:print(result,file=regtest)# alternative method to record output:regtest.write("done")You can also use theregtest_allfixture. This enables all output to stdout to be
recorded in a test function.Run the testIf you run this test script withpytestthe first time there is no
recorded output for this test function so far and thus the test will
fail with a message including a diff:$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output not recorded yet for test_squares.py::test_squares:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
done
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================This is a diff of the current outputisto a previously recorded outputtobe. Since we did not record output yet, the diff contains no lines marked+.Reset the testTo record the current output, we runpytestwith the--reset-regtestflag:$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares.out
============================== 1 passed in 0.00s ===============================You can also see from the output that the recorded output is in the_regtest_outputsfolder which in the same folder as the test script.
Don't forget to commit this folder to your version control system!Run the test againWhen we run the test again, it succeeds:$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares PASSED                                     [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
============================== 1 passed in 0.00s ===============================Break the testLet us break the test by changing the test function to compute
11 instead of 10 square numbers:deftest_squares(regtest):result=[i*iforiinrange(11)]# one way to record output:print(result,file=regtest)# alternative method to record output:regtest.write("done")The next run of pytest delivers a nice diff of the current and expected output
from this test function:$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output differences for test_squares.py::test_squares:
(recorded output from _regtest_outputs/test_squares.test_squares.out)

>   --- current
>   +++ expected
>   @@ -1,2 +1,2 @@
>   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>    done

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================Other featuresUsing theregtestfixture as context managerTheregtestfixture also works as a context manager  to capture
all output from the wrapped code block:deftest_squares(regtest):result=[i*iforiinrange(10)]withregtest:print(result)Theregtest_allfixtureTheregtest_allfixture leads to recording of all output tostdoutin a
test function.deftest_all(regtest_all):print("this line will be recorded.")print("and this line also.")Reset individual testsYou can reset recorded output of files and functions individually as:$py.test--regtest-resettest_demo.py
$py.test--regtest-resettest_demo.py::test_squaresSuppress diff for failed testsTo hide the diff and just show the number of lines changed, use:$py.test--regtest-nodiff...Show all recorded outputFor complex diffs it helps to see the full recorded output also.
To enable this use:$py.test--regtest-tee...Line endingsPer defaultpytest-regtestignores different line endings in the output.
In case you want to disable this feature, use the-regtest-consider-line-endingsflag.Clean indeterministic output before recordingOutput can contain data which is changing from test run to test
run, e.g. paths created with thetmpdirfixture, hexadecimal object ids or
timestamps.Per default the plugin helps to make output more deterministic by:replacing all temporary folder in the output with<tmpdir...>or similar markers,
depending on the origin of the temporary folder (tempfilemodule,tmpdirfixture,
...)replacing hexadecimal numbers0x...of arbitary length by the fixed string0x?????????.You can also implement your own cleanup routines as described below.Register own cleanup functionsYou can register own converters inconftest.py:importreimportpytest_regtest@pytest_regtest.register_converter_predefremove_password_lines(txt):'''modify recorded output BEFORE the default fixeslike temp folders or hex object ids are applied'''# remove lines with passwords:lines=txt.splitlines(keepends=True)lines=[lforlinlinesif"password is"notinl]return"".join(lines)@pytest_regtest.register_converter_postdeffix_time_measurements(txt):'''modify recorded output AFTER the default fixeslike temp folders or hex object ids are applied'''# fix time measurements:returnre.sub(r"\d+(\.\d+)? seconds","<SECONDS> seconds",txt)If you register multiple converters they will be applied in the order of
registration.In case your routines replace, improve or conflict with the standard cleanup converters,
you can use the flag--regtest-disable-stdconvto disable the default cleanup
procedure.Command line options summaryThese are all supported command line options:$ pytest --help
...

regression test plugin:
  --regtest-reset       do not run regtest but record current output
  --regtest-tee         print recorded results to console too
  --regtest-consider-line-endings
                        do not strip whitespaces at end of recorded lines
  --regtest-nodiff      do not show diff output for failed regresson tests
  --regtest-disable-stdconv
                        do not apply standard output converters to clean up
                        indeterministic output
...