COMBINATORIAL TESTING

Combinatorial testing is a method that can reduce cost and improve test effectiveness significantly for many applications. The key insight underlying this form of testing is that not every parameter contributes to every failure, and empirical data suggest that nearly all software failures are caused by interactions between relatively few parameters. This finding has important implications for testing because it suggests that testing combinations of parameters can provide highly effective fault detection. This article introduces combinatorial testing and how it evolved from statistical Design of Experiments approaches, explains its mathematical basis, where this approach can be used in software testing, and measurements of combinatorial coverage for existing test data. INTRODUCTION Recognizing that system failures can result from the interaction of conditions that might be innocuous individually, software developers have long used “pairwise testing”, in which all possible pairs of parameter values are covered by at least one test. However, many faults will be triggered only by an unusual combinatorial interaction of more than two parameters. A medical device study found one case in which a failure involved a four-way interaction between parameter values (1). Subsequent investigations (2, 3, 4, 5) found a similar distribution of failure-triggering conditions: usually, many were Encyclopedia of Software Engineering, Laplante. Combinatorial Testing, Kuhn Kacker Lei 1 caused by a single parameter value, a smaller proportion resulted from an interaction between two parameter values, and progressively fewer were triggered by 3, 4, 5, and 6-way interactions. These results suggest that combinatorial testing which exercises high degree (4-way or above) combinatorial interactions is necessary to reach a higher level of software assurance. If we know from experience that t or fewer variables are involved in failures for a particular application type and we can test all t-way combinations of discrete variable settings, then we can have reasonably high confidence that the application will function correctly. The key ingredient for this form of testing is known as a covering array, a mathematical object in which all t-way combinations of parameter values are covered at least once (6, 7, 8). Generating covering arrays for complex interactions (beyond pairwise) is a difficult problem, but algorithms have been developed that make it possible to generate covering arrays several orders of magnitude faster than previous methods, making up to 6-way covering arrays tractable for many applications. While the most basic form of combinatorial testing – pairwise – is well developed, it is only in the past few years that efficient algorithms for complex covering arrays – for up to 6-way coverage – have become available. New algorithms, coupled with fast, inexpensive processors, are making sophisticated combinatorial testing a practical approach that may hold the promise of better software testing at a lower cost. This article reviews the empirical and mathematical basis for combinatorial testing and explains its benefits and tradeoffs in practice. The first section reviews the development of combinatorial testing from the statistical field of Design of Experiments (DoE), differences with traditional DoE, and the empirical basis for the application of combinatorial methods to software testing. The second section reviews practical use of these methods and their application in different types of testing problems. A concluding section discusses recent developments and some active research areas. Encyclopedia of Software Engineering, Laplante. Combinatorial Testing, Kuhn Kacker Lei 2 THE DEVELOPMENT OF COMBINATORIAL SOFTWARE TESTING Combinatorial testing for software has its roots in the statistical field of design of experiments (DoE). DoE is a statistical methodology for conducting controlled experiments in which a system is exercised with carefully chosen parameter values that allow the calculation of useful statistical information about the relationship between input factors and a parameter of interest known as the response variable. Underlying every DoE plan is an assumed statistical model for the relationship between input factors and the response variable. Generally the purpose of DoE is to improve the performance of the same or similar systems. DoE is most commonly used for systems such as agricultural production, medical treatments, and industrial manufacturing which are subject to uncontrolled variables. Frequently, the effects of many test factors each having multiple test values are investigated at the same time, so combinatorial mathematics is often used in planning DoE. DESIGN OF EXPERIMENTS The body of knowledge we call DoE today is based largely on the 1920s works of Fisher and Yates (9) for agricultural experiments. In agriculture, the test factors are treatments (such as fertilizer types and amounts) which are applied to experimental units (such as plots of land) and the yield of crop (response variable) is measured. Frequently the yield from different experimental units can be quite different and since statistical conclusions are often needed for conditions where some factors in the experiment cannot be controlled. Assignment of treatments to experimental units using a mechanical randomization device made it possible to regard the effects of uncontrolled factors as random error. Thus random assignment of treatments to the experimental units made it possible to use statistical methods to interpret the data from DoE. The types of DoE plans used in agriculture were randomized block designs, balanced incomplete block designs, Latin square designs, and their extensions. In the 1940s and 1950s, DoE methods were adapted to improve performance of industrial processes and pharmaceutical development. The factors were process variables which could be classified as primary Encyclopedia of Software Engineering, Laplante. Combinatorial Testing, Kuhn Kacker Lei 3 test factors, background factors, factors held constant, and uncontrolled factors. In the 1960s, Taguchi investigated and promoted the use of DoE methods to develop robust products and manufacturing processes. Taguchi determined that to develop high quality products it was necessary to determine product parameters at which the variation in product performance from uncontrollable factors (such as environmental factors, deterioration, and manufacturing variations) was as small as possible. Taguchi used DoE to improve the main function of a product (by amplifying effect of signal factor and attenuating effect of noise factors). He promoted the use of mathematical objects called orthogonal arrays (OAs) for his adaptations of DoE methods, and his ideas became popular among manufacturers.