Package: pytest-golden
pytest-goldenPlugin forpytestthat offloads expected outputs to data filesUsage, in short(see also:example/)Install the pytest plugin:pipinstallpytest-goldenCreate a test file (e.g.tests/test_foo.py):@pytest.mark.golden_test("test_bar/*.yml")deftest_bar(golden):assertfoo.bar(golden["input"])==golden.out["output"]The wildcard selects the "golden" files which serve as both the input and the expected output for the test. The test is basically parameterized on the files.Create one or more of such YAML files (e.g.tests/test_bar/basic.yml):input:Abcoutput:NopRunpytestto execute the test(s).Whenever the function under test gets changed, its result may change as well, and the test won't pass anymore. You can runpytest --update-goldensto automatically re-populate the output.Seedetailed usage.The case for golden testingConsider this normal situation when testing a function (e.g. a function to list all words in a sentence).foo.pydeffind_words(text:str)->list:returntext.split()tests/test_foo.pyfromfooimportfind_wordsdeftest_find_words():assertfind_words("If at first you don't succeed, try, try again.")==["If","at","first","you","don't","succeed,","try,","try","again."]You wrote a basic test for that function, but it can be quite tedious to manually write out what the expected output is, especially if the output was something bigger. Sometimes perhaps you'd resort to just writing a dummy test first and copying the actual output from the failure message. And there's nothing really wrong with that, because then you'd still manually inspect whether the new output is good.With golden testingBut let's rewrite this test using "golden testing".tests/test_foo.pyfromfooimportfind_wordsdeftest_find_words(golden):golden=golden.open("test_find_words/test_basic.yml")assertfind_words(golden["input"])==golden.out["output"]Heregolden["xxx"]will be a value read directly from the associated file. Let's create that (YAML) file:tests/test_find_words/test_basic.ymlinput:|-If at first you don't succeed, try, try again.Unlike the input,golden.out["yyy"]works a little differently. Normally it will also be just an input for the test, taken from the file (and the assertion will be a completely normalpytestassertion), but in a special "update" mode it will instead accept whatever the result is at runtime and put it back into the "golden" file. Both updating and initially populating the file is done automatically with the commandpytest --update-goldens:tests/test_find_words/test_basic.ymlinput:|-If at first you don't succeed, try, try again.output:-If-at-first-you-don't-succeed,-try,-try-again.Now, when running justpytest, the test will always assert that the result is exactly equal to the expected output. Which is just how unittests work.Now you can add all of this into your source control system.Introducing a changeLet's say you're not happy that the punctuation gets clumped with the words, so you devise a different implementation for this function.foo.pyimportredeffind_words(text:str)->list:returnre.findall(r"\w+",text)You also want to add another test case for it:tests/test_find_words/test_quotation.ymlinput:|-Dr. King said, 'I have a dream.'output:-Dr-King-said-I-have-a-dreamAnd let's just turn this into aparameterizedgolden test (one test generated per each file that matches the wildcard):tests/test_foo.pyimportpytestfromfooimportfind_words@pytest.mark.golden_test("test_find_words/*.yml")deftest_find_words(golden):assertfind_words(golden["input"])==golden.out["output"]Now if we runpytest -v, we see that all is well with the new test, which gets picked up astest_find_words[test_quotation.yml], but the code changes also made it so the previous test now disagrees! You get a normal failure message frompytestitself.Normally in such situations you'd go back to the test file and edit the expected output (if you indeed expected it to change). But with this you can instead just runpytest --update-goldens, and you'll see that instead the "golden" file gets updated by itself (with no test failure). The resulting diff can then still be viewed in your source control system:--- a/tests/test_find_words/test_basic.yml+++ b/tests/test_find_words/test_basic.yml@@ -5,8 +5,9 @@ output:- at- first- you-- don't-- succeed,-- try,+- don+- t+- succeed- try-- again.+- try+- againNow you (and potentially your code reviewers) get to decide whether this diff is an acceptable one, or whether more changes are needed. You can do another iteration on the code, and the unittest will get updated as you go, and you never need to manually edit it -- just visually inspect the changes and check them in.UsagegoldenfixtureAdd agoldenparameter to yourpytesttest function, and it will be passed aGoldenTestFixtureFactory.classGoldenTestFixtureFactorygolden.open(path) -> GoldenTestFixtureCall this method on thegoldenobject to get an actual usablefixture.Thepathargument is a path to a file, relative to the calling Python test file. Teardown is done automatically when the test function finishes.@pytest.mark.golden_test(*patterns: str)Use this decorator to:avoid having to call.openand get aproper fixturedirectly as thegoldenargument of your test function andadd parameterization to your "golden" test.Thepatternsare one or moreglob patterns, relative to the calling Python test file. One test will be created for each matched file.classGoldenTestFixturegolden[input_key: str] -> AnyGet a value from the associated YAML file, at the top-level key. May raiseKeyError.golden.get(input_key: str) -> Optional[Any]Ditto, but returnsNoneif the key is missing.golden.out[output_key: str] -> AnyIn normal mode:Get a value from the associated YAML file, at the top-level key. May raiseKeyError.If--update-goldensflag is passed:Get a proxy object for the key, which, upon being compared for equality (and subsequently asserted on), marks that the "golden" file should get an updated value for this top-level key. Such updates get performed upon teardown of the fixture: the original file always gets rewritten once.golden.out.get(output_key: str) -> Optional[Any]Ditto, but when compared toNone, marks the key as deleted from the file, rather than just having the valueNone.How to...Make a custom type representable in YAMLWe will make these types known to the underlying implementation --ruamel.yaml, but let's use only the passthrough functions provided by the modulepytest_golden.yaml. It is best to apply this globally, inconftest.py.importpytest_golden.yamlpytest_golden.yaml.register_class(MyClass)(and seedetails forruamel.yaml)Alternate example if your class is equivalent to a single value:classMyClass:def__init__(self,value:str):self.value=valuepytest_golden.yaml.add_representer(MyClass,lambdadumper,data:dumper.represent_scalar("!MyClass",data.value))pytest_golden.yaml.add_constructor('!MyClass',lambdaloader,node:MyClass(node.value))Or in the particular case of subclassing a standard type, you could just drop the tag altogether and rely on equality to the base type.classMyClass(str):passpytest_golden.yaml.add_representer(MyClass,lambdadumper,data:dumper.represent_str(data))Apply a default golden file for all tests in a moduleConsider this test where we usepytest_goldenonly for storing the outputs:NOTE: These*.ymlfiles need to be manually created first, even if empty.deftest_foo(golden):golden=golden.open("stuff/test_foo.yml")assertfoo()==golden.out["output"]deftest_bar(golden):golden=golden.open("stuff/test_bar.yml")assertbar("a","b")==golden.out["output"]The test bodies are different (so applying a pattern via amarkis not applicable), but we still want to automatically assign the golden files without repeating ourselves.To do that, we can augment thegoldenfixture like this:@pytest.fixturedefmy_golden(request,golden):returngolden.open(f"stuff/{request.node.name}.yml")deftest_foo(my_golden):assertfoo()==my_golden.out["output"]deftest_bar(my_golden):assertbar("a","b")==my_golden.out["output"]Here the name of the YAML files is based on the test name. Previously the file names were manually ensured to match. So these two snippets are fully equivalent.Note that you don't even need to come up with a separate name likemy_goldenand just overwrite the originalgoldenfixture for the whole module.Seea real example of this.