Package: testflows.combinatorics
Thetestflows.combinatoricsmodule provides a convenient collection of tools
used for combinatorial testing to check different combinations of the input parameters
including calculatingcovering arraysforpairwiseandn-wisetesting using theIPOGalgorithm.Read more about:FeaturesInstallationCovering Arrays - (Pairwise, N-wise) TestingChecking Covering ArrayDumping Covering ArrayCombinationsWith ReplacementCartesian ProductPermutationsBinomial CoefficientsFeaturesProvides support for calculating the following:pure Python implementation of covering arrays based on theIPOGalgorithm that can be used with any Python test frameworkpermutations of input parameterscombinations with and without replacementcartesian product of input iterablesbinomial coefficientsInstallationWhen used withTestFlows.com Open-Source Software Testing Frameworkthetestflows.combinatoricsmodule
comes by default as a part of thetestflows.coremodule. However, if you would like to use
it with other test frameworks, it can be installed separately as follows:pip3install--updatetestflows.combinatoricsCovering Arrays - (Pairwise, N-wise) TestingTheCovering(parameters, strength=2)orCoveringArray(parameters, strength=2)class allows you to calculate a covering array
for somekparameters having the same or different number of possible values.The class usesIPOG, an in-parameter-order, algorithm as described inIPOG: A General Strategy for T-Way Software Testingpaper by Yu Lei et al.For any non-trivial number of parameters, exhaustively testing all possibilities is not feasible.
For example, if we have10parameters ($k=10$) that each has10possible values ($v=10$), the
number of all possibilities is $v^k=10^{10} = {10}_{billion}$ thus requiring 10 billion tests for complete coverage.Given that exhaustive testing might not be practical, a covering array could give us a much smaller
number of tests if we choose to check all possible interactions only between some fixed number
of parameters at least once, where an interaction is some specific combination, where order does not matter,
of sometnumber of parameters, covering all possible values that each selected parameter could have.Note:You can find out more about covering arrays by visiting the US National Institute of Standards and Technology’s (NIST)Introduction to Covering Arrayspage.TheCovering(parameters, strength=2)takes the following argumentswhere,parametersspecifies parameter names and their possible values and
is specified as adict[str, list[value]], wherekeyis the parameter name andvalueis a list of possible values for a given parameter.strengthspecifies the strengthtof the covering array that indicates the number of parameters
in each combination, for which all possible interactions will be checked.
Ifstrengthequals the number of parameters, then you get the exhaustive case.The return value of theCovering(parameters, strength=2)is aCoveringArrayobject that is an iterable
of tests, where each test is a dictionary, with each key being the parameter name and its value
being the parameter value.For example,fromtestflows.combinatoricsimportCoveringparameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}print(Covering(parameters,strength=2))CoveringArray({'a': [0, 1], 'b': ['a', 'b'], 'c': [0, 1, 2], 'd': ['d0', 'd1']},2)[
6
a b c d
-------
0 b 2 d1
0 a 1 d0
1 b 1 d1
1 a 2 d0
0 b 0 d0
1 a 0 d1
]Given that in the example above, thestrength=2, all possible 2-way (pairwise)
combinations of parametersa,b,c, anddare the following:[('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd'), ('c', 'd')]The six tests that make up the covering array cover all the possible interactions
between the values of each of these parameter combinations. For example, the(‘a’, ‘b’)parameter combination covers all possible combinations of the values that
parametersaandbcan take.Given that parameteracan have values[0, 1], and parameterbcan have values[‘a’, ‘b’]all possible interactions are the following:[(0, 'a'), (0, 'b'), (1, 'a'), (1, 'b')]where the first element of each tuple corresponds to the value of the parametera, and the second
element corresponds to the value of the parameterb.Examining the covering array above, we can see that all possible interactions of parametersaandbare indeed covered at least once. The same check can be done for other parameter combinations.Checking Covering ArrayThecheck()method of theCoveringArraycan be used to verify that the tests
inside the covering array cover all possible t-way interactions at least once, and thus
meet the definition of a covering array.For example,fromtestflows.combinatoricsimportCoveringparameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}tests=Covering(parameters,strength=2)print(tests.check())Dumping Covering ArrayTheCoveringArrayobject implements a custom__str__method, and therefore it can be easily converted into
a string representation similar to the format used in theNIST covering array tables.For example,print(Covering(parameters,strength=2))CoveringArray({'a': [0, 1], 'b': ['a', 'b'], 'c': [0, 1, 2], 'd': ['d0', 'd1']},2)[
6
a b c d
-------
0 b 2 d1
0 a 1 d0
1 b 1 d1
1 a 2 d0
0 b 0 d0
1 a 0 d1
]CombinationsThecombinations(iterable, r, with_replacement=False)function can be used to calculate
all r-length combinations of elements in a specified iterable.For example,fromtestflows.combinatoricsimportcombinationsparameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}print(list(combinations(parameters.keys(),2)))[('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd'), ('c', 'd')]Note:This function is equivalent to theitertools.combinationsWith ReplacementYou can calculate all combinations with replacement by setting thewith_replacementargument toTrue.For example,fromtestflows.combinatoricsimportcombinationsparameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}print(list(combinations(parameters.keys(),2,with_replacement=True)))[('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'c'), ('c', 'd'), ('d', 'd')]Note:Thewith_replacement=Trueoption is equivalent toitertools.combinations_with_replacementCartesian ProductYou can calculate all possible combinations of elements from different iterables using
the cartesianproduct(*iterables, repeat=1)function.For example,fromtestflows.combinatoricsimport*parameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}print(list(product(parameters["a"],parameters["b"])))[(0, 'a'), (0, 'b'), (1, 'a'), (1, 'b')]Note:This function is equivalent to theitertools.productPermutationsThepermutations(iterable, r=None)function can be used to calculate
the r-length permutations of elements for a given iterable.Note:Permutations are different fromcombinations. In a combination, the elements
don’t have any order, but in a permutation, the order of elements is important.For example,fromtestflows.combinatoricsimport*parameters={"a":[0,1],"b":["a","b"],"c":[0,1,2],"d":["d0","d1"]}print(list(permutations(parameters.keys(),2)))('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c')]and as we can see, both(‘a’, ‘b’)and(‘b’, ‘a’)elements are present.Note:This function is equivalent to theitertools.permutationsBinomial CoefficientsYou can calculate the binomial coefficient, which is the same as
the number of ways to choosekitems fromnitems without repetition and without order.Binomial coefficient is defined aswhen $k <= n$ and is zero when $k > n$For example,fromtestflows.combinatoricsimport*print(binomial(4,2))6which means that there are 6 ways to choose 2 elements out of 4.Note:This function is equivalent to themath.comb