日本ソフトウェア科学会第 31 回大会 (2014 年度) 講演論文集

When Project Centralization and Random Testing Meet–Eﬃcient Automatic Testing of Multiple Software Product Variants–

Lei MaCyrille Artho

Cheng Zhang

Hiroyuki Sato

Recently, multiple versions of a software artifact have become common occurrences in software development
and maintenance. Many similar software product variants with diﬀerent versions are created in software
evolution and Software Product Lines (SPL). This brings challenges in testing these multiple products ina cost-eﬀective way. Testing each product variant separately causes redundancies in testing the commoncode. Moreover, the test results are not easily sharable among multiple versions.
In this paper, we present a framework to test multiple product variants simultaneously. We propose theuse of project centralization to represent multiple versions as a meta-product that preserves the behavior ofeach version. We also discuss the adaptation issues of existing random testing tool Randoop to support multiple versions and coverage analysis. Furthermore, we point out the potential issues of project centralization
for more accurate and sound multi-version automatic testing.

maintenance.

According to Lehmann’s software

1 Introduction

evolution law [4], a software system requires con-Unit testing is a widely accepted and importanttinuously changes to increase its functionalities, tosoftware quality assurance technique. A unit test of

fix bugs, and to adapt to new requirements overan object-oriented program consists of a sequence

its life cycle. These version changes are released asof method invocations as inputs to test a method.

a sequence of updated versions. With the widely

Manually crafting test sequences, however, is la-adoption of revision control systems like git or mer-bor intensive. Automatic testing techniques reduce

curial, more and more software product variants are

such human eﬀorts by generating test cases auto-

created in the software evolution cycle.

Random testing [1] [8] is such an au-

The creation of multiple similar products withtomatic testing technique. It is easy to use andmatically.

diﬀerent versions is also facilitated by the modern

scalable by generating test sequences randomly. It

software development paradigm Software Productalso has been proved to be useful in detecting un-Lines (SPL) engineering [9] and the popular indus-

known bugs [8]. However, existing random testingtrial clone-and-own approach. Software producttechniques do not support to test multiple similarline engineering (SPLE) allows to systematically

products with diﬀerent versions cost-eﬀectively.

generate families of similar products to address aMultiple versions of a software artifact are com-

particular market segment or to fulfill a specificmon occurrences in software development andmission [9]. The clone-and-own approach createsa new software product by copying and modifying

Lei Ma, The University of Tokyo, Japan.
Cyrille Artho, RISEC, AIST, Japan.
Cheng Zhang, The University of Waterloo, Canada.
Hiroyuki Sato, The University of Tokyo, Japan.

an existing one.
Although many similar product variants are produced, there currently lacks a framework for the au-

tomatic testing of these products eﬃciently. Test-

run-time analysis to further guide test case gener-

ing each product separately would cause redundan-ation. However, they have not further investigatedcies in testing common code. Moreover, the in-whether the test reusing among multiple products

dividual testing results are also diﬃcult to sharewith diﬀerent versions could improve coverage.

among multiple versions to improve the overall per-Xu et al. [11] attempted to test multiple products

formance. For example, if multiple software prod-of an SPL by using test suite augmentation. They

ucts have an identical function, independently exe-

have studied the influence of the order of testing

cuting the same test case on such a function for each

multiple products on the testing performance.

product causes redundant executions, because theCompared to these works, our project centraliza-same implementation of a function shows the sametion technique [5] performs program transformation

run-time behavior in each of its occurrences.

and tests multiple products simultaneously, which

Project centralization [6] [7] transforms multiple

does not depend on the order in which products aresoftware products to a single meta-product that

tested. Combining program transformation withshares common code while preserving the behav-

random testing automatically generates test cases

ior of each product. It has been used to verify thefor multiple products without the necessity of hav-

distributed applications with multiple versions ofing to provide test cases.

multiple peers [6].
In this paper, we first summarize our preliminary

3 Project Centralization

results [5] by leveraging project centralization and3. 1 Overview of Project Centralization

random testing on multiple products from SPLs.

Project centralization [10] [6] [7] is a general tech-Then, we discuss the limitation of existing tech-nique to merge multiple software products to shareniques, and further elaborate our work in progresscommon code where possible, while resolving theimprovements of project centralization and testingversion conflicts of multiple versions [2]. It uses

strategies. Our overall goal is to build a generalprogram transformation to represent multiple soft-framework to automatically test multiple similar

ware products as a single meta-product, which pre-software products in a cost-eﬀective way.

serves the behavior of each product after centralization. In this way, the simultaneous program anal-

2 Related Workysis, verification, and testing on multiple products

Although there exists a large body of work on au-

can be simulated by the execution of the central-

tomatic test case generation, limited work has beenized meta-product. Our previous work [6] [5] hasdone for fully automatic testing of multiple version

demonstrated the eﬀectiveness of project central-

simultaneously. This section summarizes the mostization in the verification and testing of software

closely related work, which is covered in more depth

products with multiple versions.

in our previous publication [5].

In principle, the concept of project centraliza-

Several previous work has shown that the reusetion can be applied to class or method granu-of testing results could improve the testing cover-larity. Class-level project centralization performs

age. OCAT [3] reused the objects captured fromcode transformation without merging files, whilesample test cases as inputs for further test casemethod-level project centralization shares moregeneration. Palus [12] trained a method sequencecommon code by merging files while possible. As

model from the provided test cases and combined

the program behavior preservation with file merg-ing requires knowledge of underlying programming
language structure, method-level project centralization has the advantage of sharing more code, atthe expense of being more language-specific. OnP1

the other hand, class-level project centralization
can be easily generalized to diﬀerent languages.

public class A {private int field;
public void m1(int a) {this.field = a;}}public class B {private A field;
public void m1(A a) {this.field = a;}}Unit testing exercises the software under test to
explore more program paths and states to uncoverdefects. Sharing more code by file merging is desirable because it reduces redundancies in testing thesame code multiple times in each of its occurrencesin diﬀerent versions.†1

P2

public class A {private int field;
private int field_add;
public void m1(int a) {this.field = a+1;}
public void m2(int a, int b){ this.field = a;this.field_add = b;}}public class B {private A field;
public void m1(A a) {this.field = a;}}

3. 2 Testing Multiple Products of SPLs
In our previous work [5], we have investigated thebenefit of using method-level project centralizationin testing multiple software products of SPLs. We
first identify all consistent class files for merging
and rename those classes that cannot be merged
due to semantic restrictions in the underlying program language. After renaming, we merge all consistent classes. The centralized meta-product andtransformation map are then used as the input
project of our adapted random testing tool Randoop [8] that supports the version switching be-P3

public class A {private Object field;
public void m1(Object obj)
{this.field = obj;}}
public class B {private A field;
public void m1(A a) {this.field = a;}}(a). Before Centralization

図1

public class A_P1_2{
private int field;
private int field_add;
public void m1(int a) {switch (VersionSelected){ case P1.id: m1_P1(a);break;
case P2.id: m2_P2(a);break;
default: break;}}public void m1_P1(int a) {this.field = a; }public void m1_P2(int a) {this.field = a+1; }public void m2(int a, int b) {this.field = a;this.field_add = b; }}public class B_P1_2 {private A_P1_2 field;
public m1(A_P1_2 a) {this.field = a; }}public class A_P3 {private Object field;
public void m1(Object obj)
{ this.field = obj; }}public class B_P3 {private A_P3 field;
public m1(A_P3 a) {this.field = a; }}(b). After Centralization

Example of Method-level ProjectCentralization

tween multiple versions. We have also implemented
the code coverage analysis separation for each prod-tent classes in P3 and merge classes in P1 and P2 .

uct by only analyzing the centralized product. InTo merge the class A, we first merge all attributes

this way, we enable the testing multiple softwarefrom diﬀerent versions of A in P1 and P2 . Method

products eﬃciently, which improves the code cov-m2 only occurs in P2 . It has only one partitionerage by using less test generation time.

and could be directly merged as a method in theFigure 1 gives an example of method-level project

centralized code. The case for method m1 is more

centralization for three software products P1 , P2

involved, because it occurs in P1 and P2 as diﬀerent

and P3 . In this example, both classes A and B

versions. To merge m1, we first create a central-

are consistent for merging in P1 and P2 . However,ized method m1 and rename original methods tothe class A is inconsistent between P1 , P2 , and P3 ,diﬀerent names. The renamed method names are

due to the incompatible class attributes field with

m1 P1 and m1 P2 , respectively (see Fig. 1(b)).

diﬀerent types. Therefore, we rename the inconsis-

In the centralized method body of m1, we use the†1 Test sequence execution is one of most expensive
activities in the automatic test case generation.

switch statement to check out and forward the corresponding method through run-time version selec-

tion. Our transformation preserves the method be-graph cg to store the version relation of all classes

havior of each original product in the centralizedwith the same name as cl so that diﬀerent versions

meta-product for testing.

are connected by edges in cg. To centralize mul-However, our previous work has not considered

tiple products, we first initialize the corresponding

static class attributes [5]. This could cause false

D-graph. Since we also consider class file merging,positives (failed tests) when testing centralized

we mark those consistent classes with no version

products. Such test failures do not happen whenconflict and remove the corresponding edges in thetesting the original products. The reason for thisconstraint graph. Then, we start the iterative ver-is that class static attributes are unique for eachsion constraint equation solving stage until no con-class. Multiple versions of a centralized productflict edges exist. Next, we start the second renam-

would interfere each other on such attributes with-ing and re-merging transformation phase to output

out proper transformation. For class file merging,the meta-product. We process each node of the D-

our previous work renames all those classes can-graph in the topological order. We update their

not be merged and propagates the rename eﬀects

renamed references and re-check class consistency

so that the remained consistent classes are merged.

for each graph node and merge all consistent classThis loses chance to further share consistent classes,

in each node. In this way, some consistent classes

because classes that depend on them are renamed

after reference updates whose dependent classes are

diﬀerently based on internal code dependencies [7].

renamed diﬀerently could still be shared.

Moreover, we test each method equally often, with-

To favor test common methods, we measureout favoring common classes that are shared among

the importance of each method by calculating its

multiple products. In the rest of this section, weweight and use the weight to probabilistically se-discuss our in progress solutions to these issues.

lected each method from all the methods undertest. The weight formula of a method m is de-3. 3 Refinement of Method-level ProjectCentralization

fined as W eight(m) =√#Shared(m), where#Shared(m) represents the number of products

Our first refinement of method-level project cen-that share m after project centralization. A moretralization is to support static class attribute trans-advanced weight formula could also be devised

formation so that each product has its own version

when other run-time information is available, likeof the static attribute in the centralized project.

the coverage ratio of methods. Furthermore, we

Like in previous work [10], we also transform each

could balance the testing of multiple products withstatic field into an array and add one dimension if

diﬀerent versions based on the product or method

the field is an array itself. All the references of thecoverage, or user-defined weights.

static fields are also transformed correspondingly sothat we could use the project identity to run-time

4 Conclusion and Future Work

access these fields.

In this paper, we have discussed the challenges ofTo share as much code as possible, we refineautomatically testing multiple versions of software.

our project centralization further by adding the re-We have proposed project centralization as the gen-merging phase. We leverage our D-graph represen-eral framework to test the multiple software prod-tation [7] for multiple products. In the D-graph,ucts simultaneously. Essential issues of project cen-each node has a class name cl and a constraint

tralization are covered by past work, but as shownin this paper, some refinements are still needed.
Our main future work is to finish the implementation of our proposed techniques.

We are also

considering to include some program analysis techniques, such as the coverage analysis, to enhance
automatic testing of multiple products. We also
aim at evaluating our tool on a wide range of benchmarks, such as multiple versions from software evolution and SPLs.
Acknowledgement This work was supportedby the SEUT program, the University of Tokyo.
参 考 文 献[ 1 ] Hamlet, R.: Random Testing, Encyclopedia ofSoftware Engineering, Marciniak, J.(ed.), Wiley,
1994, pp. 970–978.
[ 2 ] Hnetynka, P. and Tuma, P.: Fighting class nameclashes in Java component systems, Modular Programming Languages, LNCS, Vol. 2789, Springer,2003, pp. 106–109.
[ 3 ] Jaygarl, H., Kim, S., Xie, T., and Chang,C. K.: OCAT: Object Capture-based AutomatedTesting, Proceedings of the 19th International Symposium on Software Testing and Analysis, ISSTA’10, Trento, Italy, 2010, pp. 159–170.
[ 4 ] Lehman, M. and Ramil, J.: Software evolution
in the age of component-based software engineering,
Software, IEEE Proceedings, Vol. 147, No. 6(2000),
pp. 249–255.
[ 5 ] Ma, L., Artho, C., Cheng, Z., and Sato, H.: Efficient Testing of Software Product Lines via Cen-tralization, Proceedings of 13th ACM International
Conference on Generative Programming: Concepts& Experiences, GPCE ’14, 2014 (to appear).
[ 6 ] Ma, L., Artho, C., and Sato, H.: Analyzing
Distributed Java Applications by Automatic Centralization, Computer Software and Applications
Conference Workshops, COMPSACW ’13, Kyoto,Japan, 2013, pp. 691–696.
[ 7 ] Ma, L., Artho, C., and Sato, H.: Project Centralization Based on Graph Coloring, Proceedingsof ACM 29th Annual Symposium on Applied Computing, SAC’14, Gyeongju, South Korea, 2014,
pp. 1086–1093.
[ 8 ] Pacheco, C. and Ernst, M.: Randoop: Feedbackdirected Random Testing for Java, Proceedingsof 22nd ACM SIGPLAN Conference on Objectoriented Programming Systems and Applications
Companion, OOPSLA ’07, Montreal, Canada, 2007,
pp. 815–816.
[ 9 ] Paul, C. and Linda, N.: Software Product Lines:
Practices and Patterns, Addison-Wesley Longman
Publishing Co., Inc., Boston, USA, 2001.
[10] Stoller, S. D. and Liu, Y. A.: Transformations
for model checking distributed Java programs, SPIN
’01: Proc. 8th international SPIN workshop on
Model checking of software, NY, USA, SpringerVerlag New York, Inc., 2001, pp. 192–199.
[11] Xu, Z., Cohen, M. B., Motycka, W., and Rothermel, G.: Continuous Test Suite Augmentation in
Software Product Lines, Proceedings of the 17th
International Software Product Line Conference,
SPLC ’13, Tokyo, Japan, 2013, pp. 52–61.
[12] Zhang, S., Saﬀ, D., Bu, Y., and Ernst, M. D.:
Combined Static and Dynamic Automated Test
Generation, Proceedings of the 2011 International
Symposium on Software Testing and Analysis, ISSTA ’11, Toronto, Ontario, Canada, 2011, pp. 353–363.