R1-9

SASIMI 2012 Proceedings

Accelerating Regression Test of Compilers by Test Program Merging
Takayuki Fukumoto 1

Kazushi Morimoto 2

Nagisa Ishiura 1

1

2

School of Science and Technology, Kwansei Gakuin University, Sanda, Hyogo, Japan
Securities Systems Management Services Department, Nomura Research Institute, Tokyo, Japan

Abstract—This paper proposes a method of accelerating regression test of compilers by merging test programsin compiler test suites. Large amount of computation
time is needed for compiler testing through test suites,
for they consist of a huge number of test programs. Especially, in early stages of compiler development, reductionof time for testing is a critical issue, for bug ﬁxes andregression tests are alternately repeated for many times.
The proposed method attempts to shorten the time fortest suite run by merging test programs in the test suite
into longer but fewer programs, which drastically reduces
the overhead for ﬁle open/close. During the merger, conﬂicts among the names of global variables, functions, anduser deﬁned types are avoided by preﬁxing. Header ﬁleinclusion as well as multiplier compilation are carefully
handled so that the semantics of the original test programs are maintained. A technique is also proposed to
identify test programs that resulted in execution errors
while executing the merged test programs. In an experiment where about 9,000 test programs in the testgen2 test
suite were merged into 117 programs, computation time
was reduced into 1/11.1 on Ubuntu Linux and into 1/63.9
on Cygwin on 2.5GHz Core i5 CPU.

ment, where test suites still detects a lot of errors, bug
ﬁxes and regression tests are frequently repeated, where
the number of remaining bugs reported by the test suite
works as a barometer of the progress of the debugging
tasks. In this situation, reduction of the time for test isof critical importance.
This paper proposes a method of reducing the time fortest suite run drastically by merging test programs. A
group of test programs are merged into a single program
where the main functions in the original programs areconverted into subroutine functions that are called froma new main function. Conﬂicts of the names are resolved
by renaming and ﬁle inclusion and multiﬁle compilation
are carefully handled so as to prevent the merging fromchanging the intent of the original test programs. Furthermore, a technique is also proposed to identify test
programs that resulted in execution errors while executing the merged test programs.
Experiments on the testgen2 C compiler test suite [5]shows that the run time was reduced into 1/11.1 and1/63.9 of those for the original test suite on Ubuntu Linuxand Cygwin, respectively.

II.
I.

Run Time for Test Suites for C Compilers

Introduction

Compiler reliability is a critical issue in all kinds ofsystem development, for compilers are infrastructure tools
to develop various software. Compilers must be tested
thoroughly in many ways which include testing by test
suites [1, 2, 3, 4, 5], random testing [6, 7, 8, 9], etc.
A test suite of compilers is a set of test programs which
are designed to check compilers’ functionalities. The sizeof the test suites tend to be huge, for they must test compilers with respect to a broad range of requirements fromthe language speciﬁcations. The number of test programsin a test suite may range from several thousands to millions, which results in very long run time.
The run time might not be a big issue when the test
suite is run only once. However, as long as bugs are detected, they must be ﬁxed, after which the test must be
rerun in order to verify that the bugs has been removed
and also that new bugs are not introduced by the modiﬁcation. Especially in early stages of compiler develop-

One of the most popular way of testing compilers is
to compile and execute a set of programs to see if thegenerated codes behave as expected. The programs arecalled test programs and the set of the test programs arecalled a test suite.
A compiler test suite usually consists of a huge numberof test programs. A primary reason for this is that compilers have so many functionalities to be validated. Anotherreason is that the test programs are often broken into
smaller pieces so that the debugging of the compilers will
be easier when errors are detected. This makes the runtime for test suites longer for its LOCs, due to increased
overhead for ﬁle open/close.
There are several types of test suites for C compilers.
Validation test suites [1] and commercial test suites [2, 3],
which are used for mature compilers, consist of hundred
thousands to millions of test programs. The test suite
distributed with GCC (GNU Compiler Collection) [4] is- 42 -

Ϭϭ͗ηŝĨĚĞĨ ^z^Wͺ,ϬϮ͗ηŝŶĐůƵĚĞ͞ƐǇƐĚĞƉ͘Ś͟Ϭϯ͗ηĞŶĚŝĨ
Ϭϰ͗ηŝŶĐůƵĚĞ͞ƐƚŶĚ͘Ś͟Ϭϱ͗Ϭϲ͗ŵĂŝŶ;Ϳ΂Ϭϳ͗ ƐƚĂƚŝĐŝŶƚ sĂƌŝĂďůĞ͖Ϭϴ͗Ϭϵ͗ ŝƚĞƐƚ сϬ͖ϭϬ͗ sĂƌŝĂďůĞсϭ͖ϭϭ͗ ŝƚĞƐƚ сsĂƌŝĂďůĞ͖ϭϮ͗ϭϯ͗ ŝĨ;ŝƚĞƐƚ ссϭͿϭϰ͗ƉƌŝŶƚŽŬ;Ϳ͖ϭϱ͗ ĞůƐĞ
ϭϲ͗ƉƌŝŶƚŶŽ;Ϳ͖ϭϳ͗ϭϴ͗ ƌĞƚƵƌŶϬ͖ϭϵ͗ ΃ƚϬϬϭ͘ĐϬϭ͗ ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϬϮ͗ ͬΎƚĞƐƚϭΎͬϬϯ͗ ΃ƚϬϬϮ͘ĐϬϭ͗ ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϬϮ͗ ͬΎƚĞƐƚϮΎͬϬϯ͗ ΃ƚϬϬϯ͘ĐϬϭ͗ ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϬϮ͗ ͬΎƚĞƐƚϯ ΎͬϬϯ͗ ΃ĚŝƌĞĐƚŽƌǇ ƚĞƐƚͲĂƚĞƐƚͲĂ͘Đ

Fig. 1. An example of test programs.

Fig. 2. Basic concept of test program merging.

composed of abut 3,000 ﬁles and is also used to verify thecorrectness of retargeting.
On the other hand, some test suites, such as testgen2[5], are intended for use during compiler development.
The testgen2 test suite is a collection of about 9,000 test
programs to check basic functionalities of C compilers.
Fig. 1 is an example of the test programs in testgen2. It
checks if accesses to a static variable work correctly, whereprintok() and printno() are user deﬁnable macros toreport the result1 .
During compiler development, the test suite should be
run every time bugs are ﬁxed, for it must be conﬁrmedthat the bugs have been certainly removed and that thefunctionalities which had been valid before the bug ﬁx
have not been impaired. On 2.5 GHz Core i5 CPU with
4GB memory, the run time for the testgen2 test suite isabout 7 minutes on Ubuntu Linux and 4 hours 49 minutes
on Cygwin, while the time necessary for building GCC(version 4.5.1) is about 11.5 minutes on Ubuntu Linux and39.5 minutes on Cygwin. In this situation, acceleration oftest suite run becomes a very important issue, especiallyon Cygwin where ﬁle accesses are slow.
III.
A.

Ϭϭ͗ ŝŶƚ ƚϬϬϭͺŵĂŝŶ;Ϳ΂ϬϮ͗ ͬΎƚĞƐƚϭΎͬ
Ϭϯ͗ ΃
Ϭϰ͗
Ϭϱ͗ ŝŶƚ ƚϬϬϮͺŵĂŝŶ;Ϳ΂Ϭϲ͗ ͬΎƚĞƐƚϮΎͬ
Ϭϳ͗ ΃
Ϭϴ͗
Ϭϵ͗ ŝŶƚ ƚϬϬϯͺŵĂŝŶ;Ϳ΂ϭϬ͗ ͬΎƚĞƐƚϯΎͬ
ϭϭ͗ ΃
ϭϮ͗
ϭϯ͗ŝŶƚ ŵĂŝŶ;Ϳ΂ϭϰ͗ ƚϬϬϭͺŵĂŝŶ;Ϳ͖ϭϱ͗ ƚϬϬϮͺŵĂŝŶ;Ϳ͖ϭϲ͗ ƚϬϬϯͺŵĂŝŶ;Ϳ͖ϭϳ͗ƌĞƚƵƌŶϬ͖ϭϴ͗΃

Acceleration of Test Suite Run by Test
Program Merging

Basic Idea

This paper proposes a program merging technique foraccelerating test suite run, where small test programs are
combined into larger but fewer programs so that the overhead for the ﬁle accesses will be reduced.
Fig. 2 illustrates the basic concept of the proposed
method. There are three test programs t001.c, t002.c,
1 Usually printf() is used to output speciﬁc strings to the stan-

dard output, but in early development stages where printf() is not
yet available, low level I/O functions like write() or direct accessto the memory is used instead.

and t003.c in a directory test-a. They are merged
into a single program test-a.c, where main functions in
the original test programs are renamed into t001 main,t002 main, and t003 main, and are invoked from thenew main function of test-a.c. The resulting programis functionally equivalent to the three original test programs, while it is more eﬃcient in terms of the ﬁle I/O(open/close) count. Overhead for launching the compiler
is also reduced.
Although the basic merger scheme is simple, conﬂicts ofthe variable, function and type names must be taken careof. Special treatments are also needed for ﬁle inclusion
and multiﬁle compilation.
B. Resolving Name Conﬂicts by Preﬁxing
Naive merger illustrated in Fig. 2 may cause conﬂictsof the names. Cares must be taken on the names of the
global variables, the functions, and the types (declared by
typedef statements). Fig. 3 illustrates an example of the
conﬂicts. Test programs t001.c and t002.c both declareglobal variable t var and function func. Fig. 3 (b) is a
result of naive merger where conﬂicts occur with respect
to names t var and func.
This kind of conﬂicts are resolved by preﬁxing all thenames of the variables, functions, and types in the test
programs by strings identifying the original programs.
Fig. 3 (c) shows the example where distinct names are
given to the global variables and functions.
C. Handling of Header Files
Header ﬁles are classiﬁed into two categories; the system (or standard) header ﬁles and user deﬁned headerﬁles. They must be handled diﬀerently according to theircategories.

- 43 -

Ϭϭ͗ ŝŶƚ ƚͺǀĂƌ͖ϬϮ͗ŝŶƚ ĨƵŶĐ;Ϳ΂͙΃Ϭϯ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂Ϭϰ͗ ͙Ϭϱ͗ ΃Ϭϭ͗ ŝŶƚ ƚͺǀĂƌ͖ϬϮ͗ŝŶƚ ĨƵŶĐ;Ϳ΂͙΃Ϭϯ͗ŝŶƚ ƚϬϬϭͺŵĂŝŶ;Ϳ΂Ϭϰ͗ ͙Ϭϱ͗ ΃Ϭϲ͗Ϭϳ͗ ŝŶƚ ƚͺǀĂƌ͖Ϭϴ͗ŝŶƚ ĨƵŶĐ;Ϳ΂͙΃Ϭϵ͗ŝŶƚ ƚϬϬϮͺŵĂŝŶ;Ϳ΂ϭϬ͗ ͙ϭϭ͗ ΃ϭϮ͗ϭϯ͗ ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϭϰ͗ ƚϬϬϭͺŵĂŝŶ;Ϳ͖ϭϱ͗ ƚϬϬϮͺŵĂŝŶ;Ϳ͖ϭϲ͗ ƌĞƚƵƌŶϬ͖ϭϳ͗΃Ϭϭ͗ ŝŶƚ ƚϬϬϭͺƚͺǀĂƌ͖ϬϮ͗ŝŶƚ ƚϬϬϭͺĨƵŶĐ;Ϳ΂͙΃Ϭϯ͗ŝŶƚ ƚϬϬϭͺŵĂŝŶ;Ϳ΂Ϭϰ͗ ͙Ϭϱ͗ ΃Ϭϲ͗Ϭϳ͗ ŝŶƚ ƚϬϬϭͺƚͺǀĂƌ͖Ϭϴ͗ŝŶƚ ƚϬϬϭͺĨƵŶĐ;Ϳ΂͙΃Ϭϵ͗ŝŶƚ ƚϬϬϮͺŵĂŝŶ;Ϳ΂ϭϬ͗ ͙ϭϭ͗ ΃ϭϮ͗ϭϯ͗ ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϭϰ͗ ƚϬϬϭͺŵĂŝŶ;Ϳ͖ϭϱ͗ ƚϬϬϮͺŵĂŝŶ;Ϳ͖ϭϲ͗ ƌĞƚƵƌŶϬ͖ϭϳ͗΃(a) before merger(b) naive merger(c) preﬁxingƚϬϬϭ͘ĐϬϭ͗ ŝŶƚ ƚͺǀĂƌ͖ϬϮ͗ŝŶƚ ĨƵŶĐ;Ϳ΂͙΃Ϭϯ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂Ϭϰ͗ ͙Ϭϱ͗ ΃ƚϬϬϮ͘Đ

ŝŶĐĨŝůĞ͘ŚϬϭ͗ ƚǇƉĞĚĞĨ ŝŶƚ ƉƉůĞ͖ϬϮ͗ŝŶƚ /Ŷƚ͖Ϭϯ͗ůŽŶŐ>ŽŶŐ͖ƚϬϬϭ͘ĐϬϭ͗ ηŝŶĐůƵĚĞфƐƚĚŝŽ͘ŚхϬϮ͗ηŝŶĐůƵĚĞфŵĂƚŚ͘ŚхϬϯ͗ηŝŶĐůƵĚĞ͞ŝŶĐĨŝůĞ͘Ś͟Ϭϰ͗Ϭϱ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂Ϭϲ͗ ͙Ϭϳ͗ ΃ƚϬϬϮ͘ĐϬϭ͗ ηŝŶĐůƵĚĞфƐƚĚŝŽ͘ŚхϬϮ͗ηŝŶĐůƵĚĞ͞ŝŶĐĨŝůĞ͘Ś͟Ϭϯ͗Ϭϰ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂Ϭϱ͗ ͙
Ϭϲ͗ ΃

Fig. 3. Resolution of name conﬂicts.

Ϭϭ͗ ηŝŶĐůƵĚĞфƐƚĚŝŽ͘ŚхϬϮ͗ηŝŶĐůƵĚĞфŵĂƚŚ͘ŚхϬϯ͗Ϭϰ͗ƚǇƉĞĚĞĨ ŝŶƚ ƚϬϬϭͺƉƉůĞ͖Ϭϱ͗ŝŶƚ ƚϬϬϭͺ/Ŷƚ͖Ϭϲ͗ůŽŶŐƚϬϬϭͺ>ŽŶŐ͖Ϭϳ͗ŝŶƚ ƚϬϬϭͺŵĂŝŶ;Ϳ΂͙΃Ϭϴ͗Ϭϵ͗ƚǇƉĞĚĞĨ ŝŶƚ ƚϬϬϮͺƉƉůĞ͖ϭϬ͗ŝŶƚ ƚϬϬϮͺ/Ŷƚ͖ϭϭ͗ůŽŶŐƚϬϬϮͺ>ŽŶŐ͖ϭϮ͗ŝŶƚ ƚϬϬϮͺŵĂŝŶ;Ϳ΂͙΃ϭϯ͗ϭϰ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϭϱ͗ ƚϬϬϭͺŵĂŝŶ;Ϳ͖ϭϲ͗ ƚϬϬϮͺŵĂŝŶ;Ϳ͖ϭϳ͗ ƌĞƚƵƌŶϬ͖ϭϴ͗΃ƚĞƐƚͲď͘ĐĚŝƌĞĐƚŽƌǇ ƚĞƐƚͲď

Fig. 4. Handling of header ﬁles.

The system header ﬁles are common to all the programs. Since all the functions, macro variables, and global
variables deﬁned in the system header ﬁles are referenced
from the other functions by the same name, their names
need not be be modiﬁed. Although almost all the system header ﬁles are written so that they can be included
multiple times, the same system header ﬁles should be
included only once for safety.
On the other hand, the user header ﬁles involve thealiasing problem; diﬀerent header ﬁles may deﬁne diﬀerent
variables or functions in the same name, which should be
renamed as stated in the previous subsection.
Based on the observation above, header ﬁles are handled in the following way.
1. The system header ﬁles are not expanded. If there are
multiple occurrences of inclusion for the same headerﬁle, they are reduced to one.
2. The user header ﬁles are expanded into the test programs before they are merged. During the merger,
the names of all the variables, functions, and types
are modiﬁed as described in the previous section.
Fig. 4 shows an example of header ﬁle processing.
Directory test-b contains two test programst001.c and t002.c, where t001.c includes standard
header ﬁles stdio.h and math.h along with user header
ﬁle incfile.h, while t002.c includes stdio.h andincfile.h. The program test-b.c on the right handside is the result of the merger. The two system header
ﬁles are included only once (on lines 01 and 02). Lines
04 through 06 are the expansion of user header ﬁleinclfile.h, which is the result from the include statement on line 03 of t001.c. The names are modiﬁed tohave preﬁx “t001 .” Lines 09 through 11 are also the expansion of inclfile.h but “t002 ” are preﬁxed to thenames this time.

The procedure above is easily implemented if C preprocessors can selectively handle the system and user headerﬁles. For example, gcc with -E -nostdinc option expands only user header ﬁles but ignores include statements for the system header ﬁles.
D.

Merging of Input/Output Data

If test programs are accompanied with input data andexpected output data, they must also be merged. This israther straightforward; they just need to be concatenated.
Sometimes, as is in the testgen2 test suite, the occurrencesof speciﬁc strings are counted. In that case, the sum of the
expected counts are used for the merged test programs.
E. Multiﬁle Compilation
Test suites may contain tests to validate multiﬁle compilation. In this case, the simple preﬁxing scheme described in subsection III.B. does not work, for the same
program ﬁle is used multiple times in diﬀerent combinations of other ﬁles. To solve this problem, the renaming
scheme is extended so that the preﬁxes will be the concatenations of the identifying strings of the program ﬁlesused for multiﬁle compilation.
Fig. 5 illustrates this scheme. Directory test-c has
four program ﬁles where A.c contains a main function.
A test scenario for multiﬁle compilation is assumed to
be speciﬁed in ﬁle FILESET. In this example, two test
cases, linking of ﬁles {A.c, B.c, C.c} and linking of {A.c,D.c}, are instructed. The test programs in directorytest-c are merged into three ﬁles test-c1.c, test-c2.c,test-c3.c, and a new FILESET ﬁle. The ﬁle test-i.c
contains the programs that appears as the i-th elements- 44 -

ĚŝƌĞĐƚŽƌǇ ƚĞƐƚͲĐ
͘Đ

͘Đ

Ϭϭ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϬϮ͗ ƐƵďϭ;Ϳ͖Ϭϯ͗΃Ϭϭ͗ŝŶƚ ƐƵďϮ;Ϳ΂ϬϮ͗ ͙
Ϭϯ͗΃

͘Đ

͘Đ

&/>^dϬϭ͗ŝŶƚ ƐƵďϭ΂ϬϮ͗ ƐƵďϮ;Ϳ͖Ϭϯ͗΃Ϭϭ͗ŝŶƚ ƐƵďϭ΂ϬϮ͗ ͙
Ϭϯ͗΃

͘Đ
͘Đ

@OK@
@OK@
@NG@

͘Đ
͘Đ

@OK@
@OK@

͘Đ

(a) before merger

@OK@
@OK@
@NG@
@OK@
@OK@

@OK@
@OK@
@NG@
====
@OK@
@OK@

(b) after merger(c) with delimiter

Fig. 6. Output of the test programs.
Ϭϭ͗ ǀŽŝĚͺͺͺŵĂŝŶ;Ϳ΂ϬϮ͗ ͺͺͺƐƵďϭ;Ϳ͖
Ϭϯ͗΃
Ϭϰ͗
Ϭϱ͗ŝŶƚ ͺͺŵĂŝŶ;ͿϬϲ͗ ͺͺƐƵďϭ;Ϳ͖
Ϭϳ͗΃
Ϭϴ͗
Ϭϵ͗ŝŶƚ ŵĂŝŶ;ǀŽŝĚͿ΂ϭϬ͗ ͺͺͺŵĂŝŶ;Ϳ͖ϭϭ͗ ͺͺŵĂŝŶ;Ϳ͖ϭϮ͗ ƌĞƚƵƌŶϬ͖ϭϯ͗΃Ϭϭ͗ ǀŽŝĚͺͺͺƐƵďϭ;Ϳ΂ϬϮ͗ ͺͺͺƐƵďϮ;Ϳ͖
Ϭϯ͗΃
Ϭϰ͗
Ϭϱ͗ŝŶƚ ͺͺƐƵďϭ;ͿϬϲ͗ ͙Ϭϳ͗΃Ϭϭ͗ ǀŽŝĚͺͺͺƐƵďϮ;Ϳ΂ϬϮ͗ ͙
Ϭϯ͗΃ƚĞƐƚͲĐϯ͘ĐƚĞƐƚͲĐϮ͘Đ&/>^dƚĞƐƚͲĐϭ͘ĐƚĞƐƚͲĐϮ͘ĐƚĞƐƚͲĐϯ͘ĐƚĞƐƚͲĐϭ͘Đ

Fig. 5. Merger of multiﬁle compilation tests.

in the combinations speciﬁed in the original FILESET.
test-1.c also contains the new main function to call all
the test cases. Note that all the names of the variables,
functions, and types are preﬁxed with the combination of
the name of the ﬁles that are tested for multiﬁle compilation. For example, all the variables and functions used fortest of combination {A.c, B.c,C.c} are renamed to havepreﬁx “A B C .”IV.

Identification of Error Programs

The method described so far is only eﬀective in conﬁrming that the compiler under test passes all the tests. Oncean error is detected by a merged test program, the original
tests should be run to identify the test cases in question.
In the actual compiler development, the test suite run
is repeated for many times even when the compiler fails
at many test cases, for the number of remaining (faulty)cases works as a barometer of the progress of compiler
development or debugging.
In this section, the merger scheme is extended so thatthe test programs at which the compiler under test fails
are identiﬁed by the run of the merged test program. Thisis realized by letting the merged programs output specialdelimiters to separate the outputs of the original test programs and also by making the merged program resumableeven if some test cases result in crashes.
Fig. 6 shows how the delimiter works. Suppose test two
programs are merged into one. Fig. 6 (a) shows the two
output sequences of the original programs where @NG@ in

the ﬁrst sequence indicates that the compiler has someproblem with the ﬁrst test case. However, the test programs are merged, the output becomes as shown in Fig.
6 (b), from which one cannot identify the test programthat caused the error. This problem is solved by letting
the merged test program output the delimiter to separate
the outputs of the test case, as in Fig. 6 (c).
The other problem to be considered is unexpected abortion of test caused by crash or timeout (to break inﬁnite
looping) of the generated codes or of the compiler itself.
If there is a test program that crashes in a group of programs to be merged, the resulting program also crashes,leaving the test cases after the crash unexecuted. In orderto precisely identify the programs that does not pass thetest, execution must be resumed after the abortion.
This is realized by extending both merged test programs and the test script. On ﬁnish or abortion of therun of the merged test program, the test script compares
the output of the test program with the expected values to check if all the test cases have been executed. If
abortion is detected, the test case in question is identiﬁed
and the test program is restarted. The merged test program is modiﬁed so that it can resume execution from thetest case speciﬁed by one of the command line arguments.
Fig. 7 is a modiﬁed version of the merged program. It canchoose the entry point according to the argument passed
from the command line. PRINTDIV() on lines 12 and 14
is a macro to output the delimiter.

V.

Experimental Results

A test merger script and a test execution script basedon the proposed method were implemented on top of the
testgen2 test suite. The scripts were written in Perl 5.10.1
and runs on Unix systems such as Linux, Cygwin, andMac OS X.
The merger was applied to the K&R directories of the
testgen2 test suite. About 9,000 program ﬁles in 85 directories were merged into 117 ﬁles. In the current implementation, the test programs in a directory were merged
into one test program. The reason why there were morethan 85 ﬁles was that some directories included tests for- 45 -

Ϭϭ͗ ŝŶƚ ƚϬϬϭͺŵĂŝŶ;Ϳ΂ϬϮ͗ ͬΎƚĞƐƚϭΎͬ
Ϭϯ͗ ΃
Ϭϰ͗
Ϭϱ͗ ŝŶƚ ƚϬϬϮͺŵĂŝŶ;Ϳ΂Ϭϲ͗ ͬΎƚĞƐƚϮΎͬ
Ϭϳ͗ ΃
Ϭϴ͗
Ϭϵ͗ŝŶƚ ŵĂŝŶ;ŝŶƚ ĂĐ͕ŝŶƚ ΎĂǀ΀΁ͿϭϬ͗ ƐǁŝƚĐŚ;/Ed;Ăǀ΀ϭ΁ͿͿ΂
ϭϭ͗ĐĂƐĞϭ͗ƚϬϬϭͺŵĂŝŶ;Ϳ͖ϭϮ͗
WZ/Ed/s;Ϳ͖ϭϯ͗ ĐĂƐĞϮ͗ƚϬϬϮͺŵĂŝŶ;Ϳ͖ϭϰ͗
WZ/Ed/s;Ϳ͖ϭϱ͗ ΃
ϭϲ͗ ƌĞƚƵƌŶϬ͖ϭϳ͗ ΃

TABLE I
Run time of testgen2 test suite on Ubuntu Linux.

settings
0% fail
20% fail
40% fail
20% abortion
40% abortion

w/o merge
7m03s
7m03s
7m03s
6m54s
6m49s

proposed
38s
39s
38s
56s
1m19s

acceleration
11.1
10.8
11.1
7.4
5.2

GCC 4.5.2 (i686-linux-gnu)/Core i5 2.5GHz with 4GB memory

TABLE IIRun time of testgen2 test suite on Cygwin.

Fig. 7. Revised version of merged test program.

multiﬁle compilation. 2 .
TABLEs I and II summarize the run time for the testgen2 test suite on Ubuntu Linux (GCC 4.5.2) and on Cygwin (GCC 3.4.4), respectively. The CPU was 2.5 GHz
Core i5 and was equipped with 4.0GB memory. In column “settings,” “x% fail” means that the tests fail at
x% of the programs in the test suite (this situation was
created by modifying the expected output for x% of thetest cases) and “x% abortion” means that the execution of
was aborted at x% of the test programs (this situation was
created by intentionally injecting a code to cause memory
access error into x% of the test programs). Columns “w/omerge” and “proposed” show run time for the test suite
before merger and after merger, respectively.
When the compilers passes all the tests, the test suite
run was accelerated by 11.1 times on Linux and 63.9 times
on Cygwin. Especially on Cygwin, where ﬁle accesses
are slow, reduction of run time is signiﬁcant. As rows
“20% fail” and “40% fail” show, the performance of themerged test programs was not eﬀected by the percentageof test cases that fails. However, frequent abortion of test
programs degrades the performance of the merged test
programs. However, even when 40% of the tests resultin abortion, which might be rather too pessimistic, themerged program was still faster by more than 5 times.
The time required for merger was 2 minutes 30 seconds
on Linux Linux and 2 hours 40 minutes on Cygwin. Notethat the merge have to be done only once, as long as there
is no change on the test suite.
Although the merged programs theoretically have thesame functionality as the original test programs, theymight behave diﬀerently depending potential compiler
malfunctions. In fact, there were cases where GCC 4.5.1
cross compiler for ARM passes all the original test pro2 More speciﬁcally, there are 9 directories out of 85 that include
testing of multiﬁle compilation. Linking of 2 ﬁles are required in
6 directories, and linking of 4, 10, and 15 ﬁles are required in one
directory each. Thus all the test cases are merged into (85 − 9) +2 × 6 + 4 + 10 + 15 = 117 ﬁlessettings
0% fail
20% fail
40% fail
20% abortion
40% abortion

w/o merge
4h49m48s
4h49m03s
4h48m38s
4h47m52s
4h49m11s

proposed
4m32s
4m35s
4m32s
24m28s45m16sacceleration
63.9
63.1
63.7
11.8
6.4

GCC 3.4.4 (i686-pc-cygwin)/Core i5 2.5GHz with 4GB memory

grams in the test suite but it fails on a merged program.
The cause of the error was not indentiﬁed yet, but ourconjecture is that merged programs have higher testing
ability than the original test programs. On the other
hand, no opposite case has been encountered so far wherea merged test program failed to detect errors in the original programs.

VI.

Conclusion

A method of accelerating regression test of compilers
by merging programs in test suite was presented in thispaper, by which computation time for test suite run was
drastically reduced. The scripts based on the proposed
method will be distributed under GPL2 along with the
testgen2 test suite. As another approach to accelerating
test suite run, we are now developing multicore versionsof the test script.

Acknowledgment
Authors would like to thank Mr. Nobuyuki Hikichi(TOPS Systems Corporation) for his discussion and advices on compiler development and test suite. We would
also like to thank Mr. Shohei Yoshida (AXE, Inc.) and
Mr. Soichiro Taga (now with Mitsubishi Electric MicroComputer Application Software Co., Ltd.) for his comments and continuous support for improving testgen2test
suite. We thank Mr. Ooki and all the members of IshiuraLab. of Kwansei Gakuin University for their discussion on
this research.

- 46 -

References[1] http://www.plumhall.com/suites.html.
[2] http://www.actest.co.uk/.
[3] http://www.ace.nl/compiler/supertest.html.
[4] http://gcc.gnu.org/install/test.html.
[5] http://ist.ksc.kwansei.ac.jp/ ishiura/pub/testgen2/index.html.
[6] Christian Lindig: “Find a Compiler Bug in 5 Minutes,” in Proc. ACM International Symposium on Automated Analysis-Driven Debugging, pp. 3–12 (Sept.
2005).
[7] Eric Eide and John Regehr: “Volatiles Are Miscompiled, and What to Do about It,” in Proc. 7th ACM
International Conference on Embedded Software, pp.
255–264 (Oct. 2008).
[8] Xuejun Yang, Yang Chen, Eric Eide, and John
Regehr: “Finding and Understanding Bugs in C Compilers,” in Proc. 2011 ACM SIGPLAN Conference
on Programing Language Design and Implementation(PLDI) pp. 283–294 (June 2011).
[9] Eriko Nagai, Hironobu Awazu, Nagisa Ishiura, andNaoya Takeda: “Random Testing of C Compilers Targeting Arithmetic Optimization,” in Proc. SASIMI2012, R1-10 (Mar. 2012).

- 47 -