# An Experimental Evaluation of Path-Based Product Line Integration Testing and Test Coverage Metrics
## 5 Experimental Evaluation of the Method
### Step 4: Calculating costs and benefits based on test coverage metrics defined in Section 3.2  
As a result of Step 1, a product line consisting of 155 MM-paths was created. This product line comprises 20 individual products as members. Table 3 shows the details on the number of products within the product line, the number of each type of xx-MM-path, and the estimated cost for each product based on test coverage. For P1, there are three vc+cv-MM-paths, two cv-MM-paths, one vc- & cv-MEP, and eight vv-MM-paths. In total, P1 has 24 MM-paths, including 11 cc-MM-paths. The cost for testing xx-MM-path for each product is determined using the equations outlined in Section 3, taking into account Assumption 1 through Assumption 3. Table 3 presents the results of the cost estimation for each product's testing based on the test coverage criteria. If the cc-coverage criterion is selected, only common MM-paths are covered during platform testing. Consequently, each product's testing must perform testing for the remaining MM-paths. For example, if the cc-coverage criterion is selected, testing for P1 covers only 13 MM-paths, the cost of which is calculated by multiplying 0.1, representing a cost for defining test data, resulting in 1.3. However, vv-MM-path coverage leads to more platform testing but reduces the cost of testing each product individually. With the vv-MM-path coverage criterion, all MM-paths are tested during platform testing, eliminating the need for additional testing costs for individual products, resulting in \({}^{\prime}0^{\prime}\) additional cost in the product testing. Conversely, the cost for stubs/drivers increases significantly. As another example, if cv-coverage criterion is selected, testing for P1 covers MM-paths, except for cc- and cv-MM-paths, because these two types of paths are covered  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Products**} & \multicolumn{8}{c}{**Number of xx-MM-Paths**} & \multicolumn{8}{c}{**Cost for Each Product Testing by Coverage Criteria**} \\ \cline{2-11}  & **Total** & **cc-** & **vc-** & **cv-** & **vc+cv-** & **vv-** & **cc-** & **vc-** & **cv-** & **vc-+cv** & **vv-** \\ \hline P1 & 24 & 11 & 0 & 2 & 3 & 8 & 1.3 &  
in platform testing. The last row represents the total cost of testing all the products for each coverage criterion. The results indicate that when platform testing covers a greater number of paths, the costs for testing in product testing are lower. Table 4 shows the following information at the product line level: the number of MM-paths; the estimated costs, which include the number of MM-paths to be tested in platform testing based on the selected test coverage criterion; the total costs for product testing in accordance with the coverage criterion; and the total cost estimates required for product line testing. Each cost is estimated using the appropriate equations derived from Equation (2) in accordance with the selected coverages. Figure 5 compares the costs of different test coverages with the cost of testing in a stand-alone manner. The cost of testing in the stove-piped manner is estimated using Equation (1). The benefits of these test coverages are estimated using Equation (3). The solid line indicates the cost of testing in a stand-alone manner. It is evident that the highest benefit can be obtained when using cc-MM-path coverage, cc-MM-path coverage, and cc+cv-MM-path coverage, whereas cost loss occurs when using the cv-MM-path coverage and vv-MM-path coverage. Although cv-MM-path coverage is costly, a positive benefit/cost ratio can be achieved by adopting either vc- or vc+cv-MM-path coverage.