# An Experimental Evaluation of Path-Based Product Line Integration Testing and Test Coverage Metrics
## 5 Experimental Evaluation of the Method
### Experiment Using Random Combinations  
xx-MEPs constitute the MM-path. Thus, the MM-path in SPL integration testing is a combination of these xx-MEPs. In this experiment, random combinations of possible MM-paths are generated by using four types of MEPs defined in Section 2, and experimental evaluation of the proposed method and metrics in terms of redundant testing and test coverage measurement is performed using the generated MM-paths. The results give answers to RQ1 through RQ3. Since the MM-paths of the actual SPL system are eventually included in the possible MEPs, the answers obtained through this experiment are also applied to the actual SPL system. The experiment proceeded as follows:  
_Step 1: Deriving MM-paths for a product line._  
To evaluate the cost savings of the proposed method and metrics, MEPs consisting of MM-paths for a product line are generated by combining four types of MEPs in 2-way, 3-way, 4-way, and 5-way configuration at random. These combinations of MEPs allow the duplication of MEPs.  
_Step 2: Extracting the MM-paths of each product at random from the MM-paths for a product line generated in Step 1._  
This step is repeated \"n\" times for \"n\" products, and the number of MM-paths is randomly determined as \"x\". Extraction allows for duplication. The reason for permitting duplication is that there can be multiple paths of the same type of MEPs covering different code segments.  
_Step 3: Estimating costs for both platform testing and product testing based on test coverage decisions._  
Test coverage is one of the metrics for xx-MM-path defined in Section 3.2. Cost estimates are made using the following assumptions:  
**Assumption 1** (\(\boldsymbol{C_{drug_{path}}}\)): _The value of \(\boldsymbol{C_{drug_{path}}}\) for one product is assumed to be \"1\", while the value of \(\boldsymbol{C_{drug_{path}}}\) for the entire product line is seven times as much as an individual product. The cost of making an asset reusable is widely estimated to be 150% of the cost of building the asset for a single use [26; 27]. Given the high complexity of MM-path derivation in a product line, in this experiment, the effort required to define the path is assumed to be seven times greater. Thus, the value of \(\boldsymbol{C_{drug_{path}}}\) for the product line is '7'._  
**Assumption 2** (\(\boldsymbol{C_{exec}}\)): _Most of this cost is attributed to determining test inputs. The cost is assumed to be '0.2' for platform testing and '0.1' for product testing, which is 50% of the platform testing cost, based on the same reason used to create Assumption 1. The effort required to determine test inputs in platform testing and product testing may not differ significantly. This is because a variant must be determined in order to run a test, even if it includes variability. This is a very conservative estimate, suggesting that it takes twice as much effort as the test execution effort in product testing._  
**Assumption 3** (\(\boldsymbol{C_{aux}}\)): _The development efforts for stubs or drivers are estimated at 0.08 per unit. This value is estimated to be 40% of the test execution effort (0.2). cv-MM-path and cv-MM-path require both stubs and drivers, while cv-MM-path only requires drivers. In the case of cv-MM-path, the development efforts for stubs or drivers are estimated to be 0.2 per unit due to the combinatorial complexity among variable values. In addition, cv+cv-MM-path includes cv-MEP, which requires drivers only and is calculated as '1.9 \(\times\) 0.08' instead of '2 \(\times\) 0.08'._