Package: pytest-cache-assert
pytest_cache_assertCache assertion data to simplify regression testing of complex serializable dataInstallationpoetry add pytest_assert_check --devUsageThe primary use case of this package is regression testing of large, serializable dictionaries, such as from an API under development.You may have parameterized test cases where you need to assert that the created dictionary stays the same, but you don’t want to manually generate the expected fields and values to compare. Instead you can capture a snapshot of the serialized data and cache the result then use the cached data to check for consistency in repeated test runs. The cached files should be checked into version control, which can be very useful as documentationThis package can minimize test case logic, while improving regression testing thoroughnessThis project was heavily inspired by the excellentpytest-recordingAlternativespytest-recording: this is the package I use and highly recommend for recording and replayingexternalAPI communication so that API requests only need to be made once for unit testing (i.e. recording API responses from Github's API called from a test suite)pytest-snapshot: I only found this package after already releasing a 1.0.0 version ofpytest_assert_cache. This package can be more configurable with a user-specified serializer and might be a good alternative. See their documentation for more infosnapshottest: This was another find after releasing a 1.0.0 version and would probably bea good alterantive for most userspytest-snapshotis much more configurable, has many more users, and is a better nameI really like the ability to quickly regenerate the cached files with--snapshot-updateThere is some interesting discussion on how best to handle fields that change between testssyrupy: a well designed alternative that is extensible and provides classes likeJSONSnapshotExtensionto store the cached data in VCSdirty-equals: broadly check values (i.e.assert result == {'counter': IsPositiveInt, ...}, etc.) rather than accessing and checking each field individual, which makes test easier to write and output errors easier to reviewpytest-pinned: write assertions likeassert result == pinnedwhere the results are stored in JSONtouca: more general implementation of snapshot-style testing using a cloud serviceBasic ExampleYou've created a new project calledpackage_awith one filepackage_a/source_file.pyand testtests/test_file.py"""package_a/source_file.py"""importsysfromdatetimeimportdatetimefromtypingimportAny,Dict,List,OptionalfrombeartypeimportbeartypefrompydanticimportBaseModelclassUser(BaseModel):# noqa: H601"""Example from pydantic documentation."""id:int# noqa: A003,VNE003name:str='John Doe'signup_ts:Optional[datetime]=Nonefriends:List[int]=[]@beartypedefcreate_data(name:str)->Dict:"""Arbitrary function that returns a dictionary.This demonstration uses pydantic, but any dictionary can be tested!"""returnUser(id=sys.maxsize,name=name).dict()"""tests/test_file.py"""importpytestfrompackage_a.source_fileimportcreate_data@pytest.mark.parametrize('name',['Test Name 1','Test Name 2'])deftest_create_data(name,assert_against_cache):"""Basic test of create_data()."""result=create_data(name=name)# One could manually create the expected dictionarycache={'id':9223372036854775807,'signup_ts':None,'friends':[],'name':name}assertresult==cache# ----------------------------------------------------------------------------------# Or utilize the pytest_cache_assert fixture to compare against the last cached versionassert_against_cache(result)pytest_cache_assertwill automatically create:tests/cache-assert/source_file/test_file/test_create_data-[Test Name 1].json(andtest_create_data[Test Name 2].json) for each of the parameters when first run by caching theresult. Below is the example fortest_create_data-[Test Name 1].json{"_info":[{"func_args":{"name":"Test Name 1"},"test_file":"test_readme.py","test_name":"test_create_data"}],"_json":{"friends":[],"id":9223372036854775807,"name":"Test Name 1","signup_ts":null}}The cached JSON files must be checked into version control and if needed, can be manually edited or deleted so that they will be regenerated when the test suite is next runMore ExamplesIn your cached dictionary, you may have variable values with more complex logic to verify, such as dates, UUIDs, etc. These can be selectively ignored, matched-if-null, or some other user-specified check:"""tests/test_readme_more.py."""fromcontextlibimportsuppressfromdatetimeimportdatetime,timedeltafromuuidimportuuid4importpytestfrompytest_cache_assertimportAssertRule,Wild,check_suppress,check_typedeftest_assert_against_cache_key_rules(assert_against_cache):"""Demonstrate use of `assert_rules`."""now=datetime.now()# noqa: DTZ005cached_data={'date':str(now),'nested':{'uuid':str(uuid4())},'ignored':{'a':1,'b':2},}test_data={'date':str(now+timedelta(hours=3)),'nested':{'uuid':str(uuid4())},'ignored':{'recursively':{'a':{'b':{'c':1}}}},}withsuppress(AssertionError):# Ensures that the cache file has been createdassert_against_cache(cached_data)assert_rules=[# To ignore values for 'ignored.a' and 'ignored.b', create a rule#   Here, we use the wildcard for dictionary keysAssertRule.build_re(pattern=['ignored',Wild.recur()],func=check_suppress),# Instead of suppressing, the type of data could be resolved and compared#   This is useful for datetime or UUID values where we expect variabilityAssertRule(pattern='date',func=check_type),AssertRule.build_re(pattern=['nested','uuid'],func=check_type),# Any "func" with arguments 'old' and 'new' can be used as a rule]# Without assert rules, an AssertionError is raisedwithpytest.raises(AssertionError):assert_against_cache(test_data)# But, with the custom logic, the cache assertion check will succeedassert_against_cache(test_data,assert_rules=assert_rules)Even More ExamplesFor more example code, see thescriptsdirectory or thetests.Customization (beta)Note: this feature is to be consideredbetaand may change, however, I will do my best to keep the same interfacesFor 2.0.0,pytest_cache_assertwas refactored to be more easily customizable with configuration options for not just the cache directory, but also for a way to override how files are named and to override how the cached test data is serialized and validated.With these configuration options, users or 3rd party packages can replace the default package behavior, such as changing the file format for data serialization (yaml,jsonlines, etc.) and/or specifying a different serialization logic. All configuration options are available by creating acache_assert_configfixture with the provided implementations.SeeAssertConfiginplugin.pyfor configuration options and more informationalways_write: Always write to the cached file so that diffs can be examined in the user's VCS.cache_dir_rel_path: String relative directory fromtests/. Default resolves totests/assert-cache/.cache_store: Configurable class for managing the cache representation. Default is local JSON.converters: register functions that handle conversion of unhandled types, such as pandas DataFramesvalidator: Custom validator for identifying and summarizing the deviations from the cache.importpytestfrompytest_cache_assert.pluginimportAssertConfig@pytest.fixture(scope='module')defcache_assert_config():returnAssertConfig(cache_dir_rel_path='custom/cache/dir')Project StatusSee theOpen Issuesand/or theCODE_TAG_SUMMARY. For release history, see theCHANGELOG.Planned Global Configuration OptionsThese are ideas for future options that are not currently implemented, but could be if there is enough interest:PLANNED:Provide CLI arguments likepytest-recording(request.config.getoption("--record-mode") or "none") for one-time changes to configurationPLANNED: Consider filters to prevent secrets from being cached:filter_headers=[['authorization', 'id'], ['authorization', 'cookies']](Although, you should be using a pre-commit hook and formatting the dict before passing to to the cache)TODO:Add tips from Jest on best practices-- treat snapshots as code, etc.ContributingWe welcome pull requests! For your pull request to be accepted smoothly, we suggest that you first open a GitHub issue to discuss your idea. For resources on getting started with the code base, see the below documentation:DEVELOPER_GUIDESTYLE_GUIDECode of ConductWe follow theContributor Covenant Code of Conduct.Open Source StatusWe try to reasonably meet most aspects of the "OpenSSF scorecard" fromOpen Source InsightsResponsible DisclosureIf you have any security issue to report, please contact the project maintainers privately. You can reach us atdev.act.kyle@gmail.com.LicenseLICENSE