Package: pytest-matrix
Easy fixture combinations.Instalitionpip install pytest_matrixQuickstart:importpytestfrommyprojectimportmy_func@pytest.mark.matrix(names=['arg_firs'],combs=[{'arg_first':['val_1','val_2'],},])deftest_my_fn(self,arg_first,arg_second,result):assertmy_function(arg_first,arg_second)==result@pytest.fixturedefarg_first_val_1(self):return'val_1'@pytest.fixturedefarg_first_val_2(self):return'val_2'@pytest.fixturedefarg_second(self):return'val'@pytest.fixturedefresult(self,arg_first,arg_second):# prepare expected result base on arg_first and arg_second fixture# in each test the arg_first parameter will have different valuereturn...ORimportpytestfrompytest_matriximportTestMatrixMixinfrommyprojectimportmy_funcclassMyTestCase(TestMatrixMixin):deftest_my_fn(self,arg_first,arg_second,result):assertmy_function(arg_first,arg_second)==resultMY_FN_FIXTURES_NAMES=['arg_first']MY_FN_FIXTURES=[{'arg_first':['val_1','val_2'],}]@pytest.fixturedefarg_first_val_1(self):return'val_1'@pytest.fixturedefarg_first_val_2(self):return'val_2'@pytest.fixturedefarg_second(self):return'val'@pytest.fixturedefresult(self,arg_first,arg_second):# prepare expected result base on arg_first and arg_second fixture# in each test the arg_first parameter will have different valuereturn...This will generate tests:MyTestCase::test_my_fn[arg_first_val_1|arg_second|result]MyTestCase::test_my_fn[arg_first_val_2|arg_second|result]Both examples are equal. But in class you have better scope controlling and other options.Please noticeresultfixture andarg_firstfixture. There is noarg_firstfixture definition, thearg_firstis created by py.test during test generation and you get access to current value ofarg_first, same as test function receive.Test DataEvery test function must be prefixed with ‘test_’
For every test function must be defined two class attributes. If test function is named ‘test_**my_function**’,
there must be definedMY_FUNCTION_FIXTURES_NAMESandMY_FUNCTION_FIXTURESlists.
You must define them in every class (they are not inherited).MY_FUNCTION_FIXTURES_NAMES:they are not required, it just could be little bit clearer some times, because you can choose order (the way how test name will be generated)list of names of fixtures to be combined in testyou can define fixtures, whichARE NOTdefined in test as parameter, these fixtures will be
stored in request.param and also it will be accessible by other fixturesMY_FUNCTION_FIXTURES:list of fixture combinationseach combination is dictkeys are same as inMY_FUNCTION_FIXTURES_NAMESvalues are list of fixture namefixture name is combination of parameter name and the list itemFixtures definitions:For every item inMY_FUNCTION_FIXTURESmust exists fixture. It does not have to be in same class.Fixtures names are defined inMY_FUNCTION_FIXTURES. The name si combination of key and each item in list.MY_FN_FIXTURES=[{'par':['a','b'],}]# will search for fixtures **par_a** and **par_b**WARNING:Be aware that every test has his own fixture context. This is useful when you want to access current value
of function parameter by fixture name, but can be easily overlooked.
Example:classMyTestCase(TestMatrixMixin):deftest_my_fn(self,par,result):# some testMY_FN_FIXTURES_NAMES=['par']MY_FN_FIXTURES=[{'par':['a','b'],}]@pytest.fixturedefpar_a(self):return'val_a'@pytest.fixturedefpar_b(self):return'val_b'@pytest.fixturedefpar(self):# THIS WILL NEVER BE USED IN GENERATED TESTS# the context of the generated test inject in every test to par fixture either par_a or par_b@pytest.fixturedefresult(self,par):# par is either value of par_a or par_b, it depends on testSimple FixturesThere are two simple fixtures types: String (with prefix ‘@’) and Integer  (with prefix ‘#’)importpytestfrommyprojectimportmy_func@pytest.mark.matrix(names=['arg_firs'],combs=[{'arg_first':['#1','@2'],},])deftest_my_fn(self,arg_first):assertarg_firstin(1,'2')There is no need to define fixtures arg_first_#1 (returning int(1)), arg_first_@2 (returning str(2)). It is impossible have functions (fixture definition) with these names in python anyway :).Test GeneratorThe test are generated for cartesian product of defined fixture_names.classMyTestCase(TestMatrixMixin):deftest_my_fn(self,s,b):# some testMY_FN_FIXTURES_NAMES=['a','b']MY_FN_FIXTURES=[{'a':['x','y'],'b':['i','j'],},{'a':['x','y'],'b':['k','l'],}]this will generate tests:test_my_fn[a_x|b_i]test_my_fn[a_x|b_j]test_my_fn[a_y|b_i]test_my_fn[a_y|b_j]test_my_fn[a_x|b_k]test_my_fn[a_x|b_l]test_my_fn[a_y|b_k]test_my_fn[a_y|b_l]MIXIN and inheritanceIS_MIXINYou can define tests in separate class and reuse them in multiple other class. You usually don’t want to collect these tests and run them. So you can add class attributeIS_MIXIN = Trueand tests in this class
will not be collected by pytest.If you use some of these mixins you have to define_FIXTURESfor each test. It could happen, that you won’t use some of the tests, or you do not want generate from some of the tests.SKIP_TESTYou can skip tests by writing the test name inSKIP_TESTSclass attribute.NOT_GENERATE_TESTSWrite name of test you don’t want to generate otNOT_GENEREATE_TESTSattribute. Difference between NOT_GENERATE_TESTS and SKIP_TESTS is that NOT_GENERATE_TESTS will be actually run, but they will not be paramatrize.AttributesIS_MIXIN,SKIP_TESTSandNOT_GENERATE_TESTSare not inherited from parent class.Example:classMyTestMixin(TestMatrixMixin):IS_MIXIN=Truedeftest_a(self):passdeftest_b(self):passclassRealTest(MyTestMixin):SKIP_TESTS=['test_a']NOT_GENERATE_TESTS=['test_b']classDeeperInheritanceTest(RealTest):SKIP_TESTS=['test_b']A_FIXTURES_NAMES=['par']A_FIXTURES=[{'par':['a','b'],}]@pytest.fixturedefpar_a(self):return'val_a'@pytest.fixturedefpar_b(self):return'val_b'This will skip:RealTest.test_aDeeperInheritanceTest.test_bAnd run these tests:RealTest.test_bDeeperInheritanceTest.test_a[par_a]DeeperInheritanceTest.test_a[par_b]Combination TesterSometimes you want test if you covered all combinations of specific fixtures. You can define the combinations you want to cover in class attributeCOMBINATIONS_COVER.test_combcover_fn_fx_x_y PASSEDclassTestCombinations(TestMatrixMixin):FN_FIXTURES=[{'x':['a','b'],'y':['c'],},{'x':['a'],'y':['d'],}]FN_FIXTURES_NAMES=['x','y']FX_FIXTURES=[{'x':['b'],'y':['d'],'z':['j','k']}]FX_FIXTURES_NAMES=['x','y','z']# **COMBINATIONS**COMBINATIONS_COVER=[{"fixture_names":['x','y'],"fixture_functions":['fn','fx'],}]deftest_fx(self):passdeftest_fn(self):pass@pytest.fixturedefx_a(self):pass#... rest of class with rest of fixtures (x_b, y_c, y_d, z_j, z_k)This will generate testtest_combcover_fn_fx_x_y. The prefix for combination cover test istest_combcover_followed by names of functions (test_fxandtest_fn) separated by underscore:fn_fx_and suffix are names of fixtures (their combinations we want to cover)x_y.This concrete test will find all types ofx(‘a’, ‘b’)andy(‘c’, ‘d’)fixtures, combine them([x_a|y_c], [x_b|y_c], [x_a|y_d], [x_b|y_d])and compare them with combinations manually defined in_FIXTURESconfiguration(fn: [x_a|y_c], [x_b|y_c], [x_a|y_d] and fx: [x_b|y_d]). If they are not equal, the test will fail and print all uncovered combinations. But this test will pass.test_combcover_fn_x_y FAILEDNow we added other test combination.classTestCombinations(TestMatrixMixin):FN_FIXTURES=[{'x':['a','b'],'y':['c'],},{'x':['a'],'y':['d'],}]# other configsCOMBINATIONS_COVER=[{"fixture_names":['x','y'],"fixture_functions":['fn','fx'],},{"fixture_names":['x','y'],"fixture_functions":['fn'],# **TEST ONLY ONE TEST'S FIXTURE COMBINATIONS**},]# rest of the class...This will generate two teststest_combcover_fn_fx_x_yPASSEDandtest_combcover_fn_x_yFAILED. The second test failed because combination of[x_b|y_d]is not covered inFN_FIXTURES. It will be also shown in test_result.test_combcover_fx_x_y FAILED OR PASSED according to scopeThere are two type of scopes which combcover can use when looking for all types of fixtures.classscope:default scopethe combcover will look in ALL_FIXTURESdefined in same classfunctionsscope:the combcover will look for fixture types only in these_FIXTURESfrom functions define in combcover configclassTestCombinations(TestMatrixMixin):FN_FIXTURES=[{'x':['a','b'],'y':['c'],},{'x':['a'],'y':['d'],}]FN_FIXTURES_NAMES=['x','y']FX_FIXTURES=[{'x':['b'],'y':['d'],'z':['j','k']}]FX_FIXTURES_NAMES=['x','y','z']# **COMBINATIONS**COMBINATIONS_COVER=[{"fixture_names":['x','y'],"fixture_functions":['fx'],"scope":'class',# this is not required *class* is default scope}]# rest of the class...The test will find all types ofx(‘a’, ‘b’)andy(‘c’, ‘d’)inALLfixtures, combine them([x_a|y_c], [x_b|y_c], [x_a|y_d], [x_b|y_d])and compare them with combinations manually defined inFX_FIXTURESconfiguration([x_b|y_d]). The result of the test will beFAILEDand missing combinations will be:[x_a|y_c], [x_b|y_c], [x_a|y_d]If you remove thescopekey fromCOMBINATIONS_COVERthe test will bePASSED, because combcover will be looking for only for fixtures type defined inFX_FIXTURES(x_a and y_d).classTestCombinations(TestMatrixMixin):FN_FIXTURES=[{'x':['a','b'],'y':['c'],},{'x':['a'],'y':['d'],}]FN_FIXTURES_NAMES=['x','y']FX_FIXTURES=[{'x':['b'],'y':['d'],'z':['j','k']}]FX_FIXTURES_NAMES=['x','y','z']# **COMBINATIONS**COMBINATIONS_COVER=[{"fixture_names":['x','y'],"fixture_functions":['fx'],"scope":'functions',# this is required}]# rest of the class...This combocover test will PASSTODO:[X] exclude test if test’s cls TestMatrixMixin.is_mixin == True
[X] force to define _FIXTURES and _FIXTURES_NAMES in every class, except mixin class
[X] raise error if _FIXTURES keys are not exactly same as _FIXTURE_NAMES
[ ] edit function to control use of all fixtures combinations
[X] check names of fixtures combinations are same as defined FIXTURES_NAMES
[X] allow skip tests
[X] allow not generate tests
[ ] validate sctructure of SKIP_TESTS, NOT_GENERATE_TESTS, FIXTURE_NAMES and FIXTURES
[ ] check for duplicity in _FIXTURES and COMBINATION_COVER