Package: pytest-parameterize-from-files
pytest-parameterize-from-filesMaking Pytest Parameterization Easy and Scalable![Pre-Commit][link 0]pytest-parameterize-from-filesis apytestplugin that parameterizes tests using data loaded from files using the
hookpytest_generate_tests().Introductionpytesthas a feature called parameterization that allows you to run
the same test function repeatedly using different inputs to test
multiple cases. However, managing the test data for parameterization can
be a problem. Sometimes the input data is very large or there are many
test cases, so that it is impractical to put all of the data into the
source code of the test.This plug-in loads the data for the test parameterization from separate
data files that are automatically matched up against the test functions.
Each function can have one or more data files associated with it, and
each file can contain multiple test cases. The data files can be in JSON
or YAML format.An additional issue with the basic pytest parameterization API is how
the user must provide the parameters. First all of the test case fixture
names in a list, followed by a list of lists with the values the
fixtures will take on, and then an optional list of test case id's.
Since the labels, values, and test case id's are in separate lists it
can be difficult to keep track of which fixture corresponds to which
value if you have many of them, and also which group of values
corresponds to which test id. The file structure uses a dict to keep the
test case id's, fixture names, and data values together in a way that is
easier on the human brain.FeaturesLoads data for tests from files into fixturesMultiple test data sets may be in one fileThere may be multiple data files for each testFixtures may refer to fixtures in other filesCan specify indirect parameterizationIntuitive and sane data file structureCompatibilityThis package is a plug-in forpytestand works with Python 3.9 and up.Tested withpytestversion 7.4.x, should work with any version 6.2.5
or higherTested with CPython 3.9-3.12 and PyPy 3.9-3.10While this code currently has a classifier of "Development Status :: 4 -
Beta", it is solid and well-tested. I will likely promote it to
"Development Status :: 5 - Production/Stable" after a little more
real-world usage.InstallationYou can installpytest-parameterize-from-filesfrom PyPI by usingpip:$ pip install pytest-parameterize-from-filesUsageTo use this plugin you need to make only two changes to your tests:Create the data file(s) with the proper names and formatsCallpytestwith the flag--param-from-filesAn example command line would be:$ pytest --param-from-filesThis parameter can be specified in a config file if you are running
under a CI/CD system that doesn't allow additional parameters easily.
For instance:# pyproject.toml:[tool.pytest.ini_options]addopts="--param-from-files"You can then access the data from the files via the fixtures defined in
those files. The most common usage is to manage test case inputs and
expected results. This allows the developer to change and add test cases
without making changes to the test code.Just as with pytest's basic parameterization, the test function must
have all of the fixtures in its parameter list. Otherwise, an exception
will be raised.The unit tests for this package are good examples of possible ways to
use this package. Look in the files in thetests/directory and the
corresponding files in thetests/pytester_example_filesdirectory.Data File MatchingA data file will be loaded if it matches all of the following criteria:The filename starts withdata_, followed by the name of the test
function with the prefixtest_removed. The remainder of the
filename may be any value, and is usually used to identify the tests
contained in the file.The filename must end in.json,.yaml, or.yml.The file is contained in a folder at or below the file that contains
the test.For example, for a test functiontest_foo(...)the filesdata_foo_part_1.json
data_foo_part_2.yaml
subfolder/data_foo.yamlwould all be loaded.Caution: Beware of situations where the name of one test is an
extended version of another. E.g., if you have two tests namedtest_foo()andtest_foo_bar(), a data file with the namedata_foo_bar.yamlwill be loaded forbothtests. To prevent this,
split the two test functions into two separate files in two different
directories or change the name of one of the test functions. Seetest_load_file_extended_name.pyandtest_load_separate_subdirs.pyin
the unit test files for this package for concrete examples of what might
happen and how to avoid it.Data File StructureEach data file may contain one or more sets of test data, in either yaml
or json format. The top level is a dict whose keys are the test id's.
Each test id is a dict whose keys are fixture names and whose values are
the test data. The test data may be anything, including container types
such as lists or dicts. An example input filedata_foo_bar.yamlmight
contain:test1:input_data_1:17input_data_2:3expected_result:51test2:input_data_1:-abcinput_data_2:3expected_result:-abc-abc-abcThis would parameterize into two test cases labeledtest1andtest2,
each with three fixtures,input_data_1,input_data_2, andexpected_result.Test Case Merging and ConflictsIf the same test case id is present in two different files, the fixtures
from the two files will be merged as long as a fixture with the same
name is not defined more than once for any particular test case id. For
example, for a test function namedtest_foo()with two data files:Filedata_foo_1.yaml;test_case_one:fixture_one:17Filedata_foo_2.yaml;test_case_one:fixture_two:170The function will be passed two fixturesfixture_one=17andfixture_two=170for a test case withid=test_case_one.However, if the fixture names are the same there will be a conflict
and the code that merges the test cases will raise an exception.Loading Fixtures by ReferenceAn additional powerful feature is the ability to load the value for a
fixture from another data file. You can have fixture data loaded from
another file by setting the fixture value to a specially formatted
string. It must be prefixed with two underscores and be of the format:__<Filename>:<test case id>:<fixture name>For instance a data filedata_other_check_3.yamlmight reference the
data filedata_foo_2.yamlfrom the previous section:check_functionality:input_data_1:42other_data:__data_foo_2.yaml:test_case_one:fixture_twoThis would result in two fixture values being sent into the test
function,input_data_1 = 42andother_data = 170, for a test case
withid = check_functionality. (Note that there is nothing preventing
an infinite self-referential loop although that is something that should
be avoided).Indirect ParameterizationPytest has a feature calledindirect parameterization, where
the parameter value is passed to a fixture function, and the return
value of the fixture function is then passed downstream. You can specify
that a fixture should be marked for indirect parameterization by
appending the suffix_indirectto the fixture name in the data file.
If the data file contains:test_case_1:variable_A:51variable_B_indirect:3test_case_2:variable_A:85variable_B_indirect:5the corresponding test code would be:@pytest.fixturedefvariable_B(request):returnrequest.param*17deftest_func(variable_A,variable_B):assertvariable_A==variable_BThe values for fixturevariable_Awould be passed directly totest_func(), but the values forvariable_B_indirectwould be passed
to thevariable_B()function and the return value would be passed in
as thevariable_Bparameter totest_func().Reporting IssuesIf you encounter any problems, pleasefile an issueincluding
a detailed description and (if possible) an example of the problem.ContributingSince this project is a pytest plug-in, it really does require
test-driven development. If you want to contribute a bug fix or new
feature, please first create a test case that demonstrates what your new
code is supposed to do. Note that you need to set things up using thepytesterfixture, rather than testing directly.This project useshatchfor its environments and build system,
as well aspre-commitandrufffor formatting and
linting. Before you send in a pull request, please:Set uppre-commitand use it to runruffandmdformatwith the
settings included in thepyproject.tomland.pre-commit-config.yamlfilesRun tests using the commandhatch run test:test, which will run all
of the tests against CPython 3.9-3.12 and PyPy 3.9-3.10Check test coverage withhatch run covLicenseDistributed under the terms of theMITlicense,pytest-parameterize-from-filesis free and open source software.ColophonInspired by the pytest plug-inspytest-datadir,pytest-datafixtures,
andpytest-xpara. I also looked at the packageparameterize-from-file.I wanted to load data from files, butpytest-datadirandpytest-datafixturesrequired code in the test specifically to read
in the file.I liked the way thatpytest-xparaloaded data into a fixture, but
didn't like that it would only work with one file and that I had to
specify the file on the command line.After I wrote much of this project I found the packageparameterize-from-fileswhich has a similar name. It's not to my taste
as I think it's trying too hard. Placing code snippets into the data
files would be a maintenance problem down the road and the process to
specify mulitple data files is clunky. Having to import the package in
every test file and decorate each function increases the complexity of
the test code. Even so, I will be renaming this project shortly so that
there will be less of a chance for confusion between them.Thispytestplugin was developed using a skeleton generated bycookiecutteralong with thecookiecutter-pytest-plugintemplate, then extensively
modified to bring it up to modern standards.