A Test Case Recommendation Method Based on
Morphological Analysis, Clustering andthe Mahalanobis-Taguchi Method
Hirohisa Aman
Center for Information Technology
Ehime University
Matsuyama, Ehime 790-8577, Japan
Email: aman@ehime-u.ac.jp

Takashi Nakano
Hideto Ogasawara
Toshiba Corporation
Kawasaki, Kanagawa 212-8582, Japan

Abstract—This paper focuses on the content of test cases,
and categorizes test cases into clusters using the similarity
between test cases; their degree of similarity is obtained througha morphological analysis. If there are two similar test cases,
they would test the same or similar functionalities in similar
but different conditions. Thus, when one of them is run for aregression testing, the remaining one should be run as well, inorder to reduce a risk of overlooking regressions. Once a test
engineer decides to run a set of test cases, the method proposedin this paper can recommend adding similar test cases to theircandidate set. The proposed method also considers the priorities
of recommended test cases by using the Mahalanobis-Taguchi
method. This paper reports on an empirical study with anindustrial software product. The results show that the proposed
method is useful to prevent overlooking regressions.

I. I NTRODUCTION
The successful release of a software product requires wellperformed testing. Since a product is usually evolved throughvarious upgrades, regression testing plays a significant role ateach release of the product. Regression testing is also a costly
activity [1] in the software development and maintenancephase—it is especially common in software products embedded in equipment: we have to install an upgraded version of
the software into a piece of equipment, and test its behaviors
by hand. Thus, to enhance the efficiency of regression testingis a major challenge for all test engineers.
When a software product is upgraded, it is not sufficient totest only the modified functions because a code modification
may cause an unexpected failure. In other words, any upgrade
has a risk of creating a regression. Therefore, it would be
useful to test not only the modified functions but also otherfunctions which might be affected by the modification. Thismeans that we need to single out possible test cases fromour test case pool, and rerun those test cases as well [1], [2].
In this paper, we propose a novel method for recommending
related test cases during a regression testing, and examine its
usefulness through an empirical study.
The key contributions of this paper are as follows: (1) weleverage a morphological analysis method for a clusteringof test cases; (2) we propose a test case recommendation

Minoru Kawahara
Center for Information Technology
Ehime University
Matsuyama, Ehime 790-8577, Japanmethod using the cluster information, in order to reduce therisk of overlooking regressions; (3) the usefulness of the
proposed method is examined through an empirical study withan industrial software product.
The remainder of this paper is organized as follows: Section
II describes studies related to this paper. Section III explains
our proposal and Section IV reports on an empirical study.
Finally, Section V presents the conclusion of this paper andour future work.
II. R ELATED W ORKWhen test engineers run test cases as a regression test, there
may be the risk of overlooking latent failures. In order toreduce such a risk, it is important to rerun as many test cases
as possible. However, there is a limit to our available effort
for testing. In this context, the prioritization of test cases is
one of the most popular topics in the software testing [1], [2].
Jeffrey et al. [3] and Mirarab et al. [4] focused on the relationship between a test case execution and its corresponding
program, and proposed ways of prioritizing test cases throughthe program slicing analysis or the code coverage analysis.
Kim et al. [5] focused on the testing history data rather than
the source program, and prioritized test cases by using thenotion of the exponentially smoothed moving average. Amanet al. [6], [7] evaluated test cases by using their failure rates in
the past and the length of the non-tested version’s sequence,and proposed considering a prioritization of test cases as a 0-1
programming problem.
There have also been studies leveraging a clustering algorithm. Sherrif et al. [8] analyzed the change history of source
files in order to classify related test cases into a cluster, andproposed prioritizing test cases using that cluster information.
Carlson et al. [9] and Leon et al. [10] categorized test cases
by using the code coverage data or the execution profiles, andprioritized test cases based on those clusters.
While the above studies seem to be useful proposals orreports, they require a linking of test cases to corresponding
programs or an accumulation of testing data; if we have little
data on test runs, those methods might not work successfully.

Thus, we believe that it would also be important to analyze
test cases from another perspective other than the test caserunning. Arafeen et al. [11] proposed performing a clusteringof test cases by focusing on the requirement specification of
the software product. Although it would be a promising way,
we have to make a link between a specification and a test
case, and such linking is not easy work. There have also
been studies focusing on the text of test cases. Ledru et al.
[12] proposed a test case prioritization method using string
distance—it incrementally builds the set of test cases to be run
by selecting the farthest test case from the set of preceding test
cases. While it is a remarkable work to prioritize test cases andmay also be utilized for clustering test cases, their method isbased on character-level distances, so it evaluates relationships
among test cases from a lexicographical perspective. That is tosay, it is better to take into account semantical aspects of test
cases as well. Thomas et al. [13] leveraged the topic modelingtechnique to quantify similarities (or distances) between test
cases. They extracted “topics” from the set of test cases, andrepresented the features of test cases as vectors of membership
degrees to extracted topics. While the work by Thomas etal. is a noteworthy preceding study, it is a different approachthan a test case clustering. From a similar perspective, we willpropose another, easier, way of test case clustering by focusing
on words of test cases in the following section.
III. T EST C ASE R ECOMMENDATION

Fig. 1. Image structure of our proposal: blue hatched circles are manuallyselected test cases, and orange ones are automatically-recommended test cases
which are similar to manually-selected ones.

Test engineers usually decide which test cases they rerun,
based on their knowledge in regard to the upgraded functions
and the past records of testing. We do not criticize such a
decision for regression testing. Since they would have specificknowledge and experience in testing their software products,
it would be a practical way of making a decision. However,there is a risk of overlooking something because test engineers
are also human beings. Thus, our aim is to automatically
recommend appropriate test cases to reduce such a risk. That
is to say, once a test engineer has decided to run a set of test
cases, our method recommends adding other test cases to theircandidate set if there are any that are appropriate (see Fig. 1).

A. Our Current Condition and Goal

B. Similarity between Test Cases

At first, we will explain the current condition of ourregression testing.
For a software product, we have n test cases, ti (fori = 1, . . . , n). The product has been evolved through minor
upgrades, and the latest version is the m-th version. For thesake of convenience, we denote the j-th version by vj (forj = 1, . . . , m); that is to say, the latest version is vm .
Now, we are about to test vm of the product. In other words,
the tests for v1 , v2 , · · · , vm−1 are already finished, and wehave the results (the test history) like those shown in Table I.
In the table, symbols “P” and “F” show that the result of thecorresponding test case’s run at the corresponding version was“Pass” and “Fail,” respectively; the symbol “—” signifies thecorresponding test case was not executed at the corresponding
version. Since we are about to test at vm , the cells in thecolumn of vm are not decided: they are denoted by “?.”
Next, we describe our goal in this study.

In general, test engineers prepare many test cases and run
them to test the functionalities of their software product. In
those test cases, there may be ones that are similar to eachother. For example, suppose we want to test two different
functions, A and B, under a certain condition, and have twotest cases tAB and tBA : while tAB executes in the order (A,B), tBA does in the opposite order (B, A). In such a case, thecontents of tAB and that of tBA would be similar. However,since they are not identical, we should not skip over eithertAB or tBA ; we have to run both of them. Some test engineers
might run only tAB , and overlook a failure which would havebeen detected by tBA .
In this paper, we focus on the similarity of the contents between test cases. Using a morphological analysis, we evaluate
their similarity. Then, when a test case is run, we consider
whether to recommend running similar test cases as well.
We describe a quantification of the similarity between test
cases in the remainder of this subsection.
1) A morphological analysis of a test case:For each test case, we prepare a plain text file describing
the content of the test case. Then, we apply a morphological analysis tool to the file, and extract words written
in each test case and their parts of speech.
Since our (the authors’) test cases are written in Japanese,
we use MeCab1 which is the most popular morphologicalanalysis tool for Japanese, as the tool.

TABLE IF ORMAT OF OUR TEST HISTORY DATA .

t1
t2
t3
.
..
tn

v1
P
—
F
.
..
—

(P: Pass;

v2
—
F
P
.
..
—

v3
—
P
—
.
..
P

F: Fail;

···
···
···
···
.
..
···

vm−2
—
F
P
.
..
—

—: No run;

vm−1
—
P
—
.
..
F

vm
?
?
?
.
..
?

?: to be decided)

1 http://taku910.github.io/mecab/

The Jaccard index shows how many words are common
to two documents (test cases). If their word sets are thesame, the Jaccard index is 1; if their word sets have nointersection, the index is 0.
2

D. Prioritizing Test Cases by Mahalanobis-Taguchi Method
While the above clustering method can form a set of test
cases that are recommended to be rerun, we also have to decide
the order of their rerunning.

0.4
0.0

Next, we categorize test cases into clusters by using thesimilarity between test cases discussed above. By clustering
similar test cases, we can narrow down a set of candidate test
cases to be recommended for regression testing; when a test

0.2

C. Clustering Test Cases

distance

0.6

Let us consider a simple example. Suppose we have threetest cases t1 , t2 and t3 : their word sets W1 , W2 and W3 areshown in Table II.
Then, we obtain the followings:• W1 ∪ W2 = {account, button, chronological, click,configuration, date, display, download, file, log, order,
password, reset, user} and W1 ∩ W2 = {button, click,file}; |W1 ∪ W2 | = 14 and |W1 ∩ W2 | = 3.
• W1 ∪ W3 = {archive, button, click, check, chronological,
date, display, download, file, order, log} and W1 ∩ W3 ={button, click, chronological, date, download, file, order};|W1 ∪ W3 | = 11 and |W1 ∩ W3 | = 7.
• W2 ∪ W3 = {account, archive, button, click, chronological, configuration, date, download, file, order, password, reset, user} and W2 ∩ W3 = {button, click, file};|W2 ∪ W3 | = 13 and |W2 ∩ W3 | = 3.
Therefore, their similarities are as follows:• J(t1 , t2 ) = 3/14 ≃ 0.214;• J(t1 , t3 ) = 7/11 ≃ 0.636;• J(t2 , t3 ) = 3/13 ≃ 0.231.

TABLE IIE XAMPLE OF WORD SETS .
word set
W1
W2
W3

words (only nouns, adjectives and verbs)
button,click,
chronological, date,display,
download, file,
log,
order
account, button,
click,
configuration,
file,
password,reset,
user
archive,button,click,
chronological,
date,
download, file,
order

265
296 251266
267 257
260
253
262
276
254
274
275
158
161
159
162
157
160 129132 130
152
155
133
151
154
143 137
146
140
131
153
134
156
141
144
135
138 142
136
139
145
294
295
149
148
270
280 150147
268
269
21
212
34
215
32
213
33
214
48
47
237
236
238
46
45
235
43
44
41
232
233
231
234
39
42 40
38
228
229
227
230
208
3543
209
2
297
51
273
271
272 264
263
59
92
65
98
62
95
60
93
91
128
71
104
76
109
74
107
72
105
75
108
63
96
64
97 114115 113
112
68
101
69
102
67
100
66
99
86
123
83
120
84
121
73
106
78
111
85
122
80
117
82
119
79
116
81
118
77
110
89
126
61
70
94
103
90
127
87
124
88
125
243
241
242
258
259 282
277
278
279
255
256
250 211
239
240
248
249
244
245
246
247
293284
288
1
285
24
281
31
29
30
25
22
23
27
26
28
36
37
299
300298 219
216
217 218 220221222
224 226225
261
50 22354 49
252
52
53 55 57
56
58
210
205
206 207
286
287
292
283
290 289
291

(1)

(2)

d(ti , tj ) is called Jaccard distance. While there are also otherdistance functions (other than Jaccard distance), we will use
it because it is easy-to-implement and widely known in thenatural language processing world.
Once a distance between test cases is defined, we can apply
a clustering algorithm to a set of test cases. In this study, we
adopt the complete linkage method as our clustering method,which is one of the most popular clustering methods, becauseof its ease of implementation. A popular open source statisticalcomputing environment R2 provides function hclust which
performs a hierarchical cluster analysis with the complete
linkage method. We can obtain a tree diagram (dendrogram)
like one shown in Fig. 2. Then, we can get cluster databy setting its “cut level,” which is the threshold of distance
between test cases, to decide if those test cases belong to thesame cluster or not. We empirically set 0.3 as the cut level inthis study.

176
177
175
173
174
202
203
204
191
192
190
188
189
17
196
193
15
20
16
195
19
199
194
197
111
8 200
198
201 178
14
187
182
184
183
185
186
13
180
12
179
181
163
172105
9
169
7
165
166
170
171
164
167
1688 6|Wi ∩ Wj |.
|Wi ∪ Wj |d(ti , tj ) = 1 − J(ti , tj ) .

1.0

J(ti , tj ) =case t is rerun as a regression test, we may recommend runningsimilar test cases as well, from the cluster which t belongs to.
In order to perform a clustering algorithm for test cases, wehave to define a “distance” between test cases. In this paper,
we will consider a more similar pair of test cases to be a closer
pair. Thus, we define the distance between two test cases ti
and tj , d(ti , tj ), as:

0.82) A set of words of interest:
In accordance with many cases of natural language processing, we focus only on a few parts of speech in thisstudy—nouns, adjectives and verbs. Then, we form theset of words (nouns, adjectives and verbs) appearing ineach test case. For the sake of convenience, let Wi be theset of words of test case ti .
3) A similarity computation:
We define the similarity between test cases ti and tj by
the following Jaccard index [14], J(ti , tj ):test case id
hclust (*, "complete")

Fig. 2. Example of dendrogram with the cut level 0.3.
2 https://www.r-project.org/

We have used the following two criteria for evaluating test
cases in the past [7]:• Gap between the Last-run version and the Current version(GLC): GLC(t) is the number of consecutive versions inwhich test case t was not run until the current version.
A higher GLC value means that the test case has not beentested for a longer time. Thus, there is a higher risk ofoverlooking a regression.
• Failure Rate (FR): FR(t) is the failure rate of test case t.
That is to say, it is the number of versions failed by t,divided by the number of versions at which t was run.
A test case with a higher FR value has a better trackrecord of detecting failures. Thus, it seems to have ahigher worth to be rerun.
In order to consistently integrate different evaluations ofa test case by the above criteria, we have leveraged theMahalanobis-Taguchi method [7], [15].
In the remaining part of this subsection, we briefly describe
how to prioritize test cases by using the Mahalanobis-Taguchi
method (see [7] for more detail).
Suppose a feature of an object is expressed by ndimensional vector x, and similar objects form a set (group)in the vector space. Let x be the mean vector of the feature
vectors belonging to a group, and let S be the variancecovariance matrix of them. Then, the following dM (x, x) is
known as the “Mahalanobis distance” between x and x:√(x − x)T S −1 (x − x) .
(3)dM (x, x) =In the simple one-dimensional case, it is:√1|x − x|,dM (x, x) =(x − x) 2 (x − x) =
σ
σ

(4)

where σ is the standard deviation; σ 2 is the variance.
That is to say, the Mahalanobis distance can be interpreted
as the “Euclidean distance” normalized by the data dispersion.
A larger Mahalanobis distance means that x has loss of amembership in the group. The Mahalanobis-Taguchi method
computes the Mahalanobis distance from the center of objects
which are working normally, and uses the distance as theevaluation of the abnormality; it is a well-known quality
control method (see Fig. 3).
Based on the notion of the Mahalanobis-Taguchi method,
we have proposed to evaluate the rerunning value of a test caseFig. 3. Example of the Mahalanobis-Taguchi method.

TABLE IIIS IMPLE EXAMPLE OF TEST HISTORY DATA , GLC, FR AND RERUNNING
VALUES OF TEST CASES .

t1
t2
t3
t4

v1
P
—
F
F

v2
—
F
P
P

(P: Pass;

v3
—
P
—
F

v4
—
F
P
P

F: Fail;

v5
—
P
—
—

v6
?
?
?
?

—: No run;

GLC
5
1
2
2

FR
0.00
0.50
0.33
0.50

value
9.16
8.46
8.03
10.25

?: to be decided)by using the Mahalanobis distance from the least valuable test
case such that both of its GLC value and FR value are zero
[7].
Table III shows a simple example of 4 test cases t1 , · · · ,t4 and 6 versions v1 , · · · , v6 : we are able to test the productat v6 . The GLC values and FR values of t1 , · · · , t4 are also
given in the table. For example, since the last-run version of
t1 is v1 , its GLC value is 5(= 6 − 1); t3 was run 3 times andit failed once (at v1 ), thus its FR value is 1/3 ≃ 0.33. Since
the covariance of GLC and FR is −0.389, and the variancesof GLC and FR are 3 and 0.056, respectively, we have thefollowing variance-covariance matrix S:][
3 −0.389
S=.
−0.3890.056
Then, we can obtain its inverse matrix S −1 :[]3.36 23.32−1
S =.
23.32 179.87
Therefore, we can compute the rerunning value of t1 with
Eq.(3) as
√
[
][
]
[
]
3.36 23.32
5−0
5−0 0−0≃ 9.16 .
23.32 179.87
0−0Similarly, we can get the rerunning values of t2 , t3 and t4 .
The computation results are shown in “value” column of Table
III.
E. Recommendation Procedure
Now, we suppose that a test engineer decides to rerun sometest cases as a regression test on a version of a software
product. Let R0 be the set of those test cases.
Then, we propose the following method for recommending other test cases in order to prevent an overlooking ofregressions; let R1 be the set of those recommended test cases(R0 ∩ R1 = ∅):1) We classify test cases into clusters by using the similarityof test case descriptions (see Sections III-B, III-C).
2) Let R1 = ∅. Then, we iterate the following operation foreach t ∈ R0 :R1 ← R1 ∪ (C(t)\R0 ), where C(t) is the cluster towhich t belongs, and “\” signifies the operator to obtain
the difference set3 .
3 x ∈ A\B ⇐⇒ x ∈ A ∧ x ̸∈ B.

3) For each t ∈ R1 , we collect its GLC value and its FR
value from its test history, then compute its rerunningvalue (the Mahalanobis distance from the least valuable
case: GLC = FR = 0) (see Section III-D).
4) We recommend t ∈ R1 in descending order of thererunning value, except for the ones whose values are
zero.

TABLE IVR ESULTS OF R ECOMMENDATIONS AND R EGRESSION D ETECTIONS .
Version
v2
v3
v4
v5
v6
v7
v8
v9
v10
v11
v12
v13

2
IV. E MPIRICAL S TUDY
A. Aim and Dataset
The aim of this empirical study is to show the usefulnessof the proposed method, by using the regression testing dataon an actual software product.
In this study, our experimental subjects are 300(= n) test
cases for a software product (a Web-based information system)that has been maintained by two of the authors. All test cases
are written in Japanese (natural language)4 . Thirteen (13(=m)) versions of the product have been tested by those 300 test
cases—for the sake of convenience, we denote these versions
by v1 , v2 , · · · , v13 in chronological order.
Since it is past test data, a follow-up investigation ofregressions is already finished. According to the results ofthe follow-up investigation, there were regressions when the
product was upgraded from v6 to v7 . However, regression
testing (rerunning test cases) missed those failures. We try
to detect those overlooked failures with the test cases which
are recommended by the proposed method.
B. Procedure
This empirical study is conducted in accordance with thefollowing procedure:1) We perform a morphological analysis on each of the
300 test cases. Then, for each combination of test cases,
we compute the distance between them based on theirsimilarity, with Eqs. (1), (2). Using those distances, weclassify the test cases into clusters. In the clustering
algorithm, we empirically set 0.3 as the cut level.
2) We iterate the following for each version (vj for j =2, . . . , 13):a) Let R0 be the set of test cases which were run at vj .
b) We create the set of recommended test cases, R1 ,according to the procedure described in Section III-E.
c) We recommend test cases belong to R1 in descendingorder of their rerunning values which are computed by
the Mahalanobis-Taguchi method.
d) We examine if the recommended test cases can detecta regression.
C. Results and Discussions
Table IV shows the results of test case recommendation ateach version:
4 Since the contents of test cases are confidential, we will omit details ofthem and avoid giving examples in this paper.

#run(|R0 |)
160
17
27
10
5
7
2
5
9
13
6
3

#recommended(|R1 |)
13
16
0
1
5
15
0
1
4
1
5
1

#defects
detected
0
0
0
0
0
6
0
0
2
0
2
0

#run column: the number of test cases that were run atthe version (past records). That is |R0 |.
• #recommended column: the number of test cases that arerecommended by the proposed method. That is |R1 |.
• #defects detected column: the number of test cases thatare included in R1 and can detect a regression. That is
to say, at a version whose #defects detected > 0, if we
also rerun those test cases, we would have detected aregression which was in fact overlooked.
Since there were regressions when the product was upgraded from v6 to v7 , #defects detected = 0 for v2 , · · · ,
v6 .
•

2
Table IV shows interesting results. At v7 which is soon after
the introduction of latent faults (regressions), only 7 test cases
were rerun (see v7 row, #run column in Table IV). However,the proposed method recommends 15 test cases to rerun inaddition to those 7, and 6 out of the newly recommended 15
test cases succeed in detecting regressions. It is a remarkable
point that the proposed method recommends more than twotimes as many test cases as the number actually run (15 vs. 7).
That is to say, the original regression testing plan at v7 seemed
to be insufficient, and it caused an overlooking of regressions.
We consider the effectiveness of the proposed method to detect
regressions at v7 as well. The proposed method recommended
15 test cases at v7 , and 6 out of them found regressions. If we
randomly selected 15 test cases (except for those 7 test cases
which were already selected) at v7 , the expected number oftest cases which succeed to detect regressions is about 1.13.
Thus, the proposed method could be about six times more
effective than the random selection.
Figure 4 shows the ratio of the recommended test cases
to the run test cases (|R1 |/|R0 |). After the occurrence oflatent regressions, the proposed method succeeds in findinga regression when the ratio is relatively high.
From these results, we can expect that our recommendation
method based on the similarity between test cases’ descriptions
is useful to prevent overlooking regressions. When a test
case’s description is similar to another’s description, those
two test cases would test similar functions or related functions

case ti and the cost to rerun ti , respectively (for i = 1, . . . , n).
Then, our test case selection problem can be formulated as:
Maximizesubject to

n
∑
i=1
n
∑

ri · xi ,ci · xi ≤ cL ,i=1xi ∈ {0, 1} (i = 1, . . . , n),Fig. 4. Ratio of the recommended test cases to the run test cases (|R1 |/|R0 |).

in partially different conditions. In such a case, it would beinsufficient to run only one of them; by skipping a run of
the remaining test case, we might overlook a regression. Thisempirical study would show such a case of oversight.
Next, we consider the usefulness of prioritizing test cases
within the recommended test case set (R1 ). As shown in Table
IV, 15 test cases are recommended at v7 . Table V shows
the results of prioritizing those 15 test cases in descendingorder of the rerunning value (the Mahalanobis distance fromthe least valuable case). In the table, test cases’ names are
relabeled by the order; the “detecting defect” column shows if
the corresponding test case succeeds in detecting a regression.
Since all of the successful test cases are ranked in the upper
half, our prioritization leveraging the Mahalanobis-Taguchi
method seems to work successfully.
Meanwhile, there might be a case that the proposed method
recommends too many test cases to run. In such a situation, we
have to single some test cases out from the recommendation
set. The simplest way is the greedy method in which we
continue to select test cases in descending order of priority
until we exhaust our testing effort. However, the greedy
method cannot present the optimal solution; there is a better,more cost-effective way in which we can formulate the test
case selection problem as a 0-1 programming problem and use
its solution [7]—let ri and ci be the rerunning value of test

TABLE VM AHALANOBIS DISTANCES OF RECOMMENDED TEST CASES AT v7 .
Test case
tx1
tx2
tx3
tx4
tx5
tx6
tx7
tx8
tx9
tx10
tx11
tx12
tx13
tx14
tx15

rerunning value (Mahalanobis distance)
8.21
8.21
8.21
8.21
8.21
8.21
8.21
8.21
8.21
6.57
6.57
6.57
6.57
6.57
6.57

detecting defect
Yes
No
Yes
No
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No

where xi expresses whether we select ti (xi = 1) or not (xi =0), and cL is the upper limit of cost which we cannot exceed.
The solution of the above problem provides the best set of test
cases to be rerun under conditions that the rerunning valuesof all test cases and the upper limit of cost are given.
Finally, we discuss the “cut level” at our clustering in thisempirical work. While we set 0.3 as the cut level based on
experience, another cut level would produce another result.
For version v7 , we examined all cut levels from 0.1 to 0.9
with 0.1 increases. Table VI shows the results: the numberof recommended test cases (#recommended), the number ofrecommended test cases which can detect a defect (#defects)
and the detection rate (= #defects / #recommended) for eachcut level. Needless to say, a higher cut level produce larger
clusters, thus more test cases are recommended, and more
regressions (defects) are also detected. While the cut level that
we empirically used (0.3) showed the best detection rate inthese candidates, it does not succeed to detect all overlooked
regressions. Thus, another cut level may be the best in anothercontext. A further analysis of cut level is our significant futurework.
D. Threats to Validity
Our findings may be subject to the following concerns.
Since our study covers a part of regression testing for asingle product, we cannot say our results are generalizable.
However, we believe this study could contribute to stir up
a utilization of the morphological analysis in the regression
testing world.
Since the test cases that we used in our empirical work
are written in a natural language, there might be a variety
of vocabulary. That is to say, different engineers might use
different words to describe the same thing in their test cases,
and they could cause incorrect evaluations of similarities

TABLE VIR ESULTS OF R ECOMMENDATIONS AND R EGRESSION D ETECTIONS AT v7
FOR DIFFERENT CUT LEVELS .
cut level
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9

#recommended
4
6
15
28
27
40
59
66
116

#defects
0
0
6
8
8
10
12
12
22

detection rate
0
0
0.400
0.286
0.296
0.250
0.203
0.182
0.190

among test cases. In order to avoid such a variety, we need toprepare a common dictionary of terms (words). Alternatively,
it would be better to perform a data preprocessing to linka word with another word which has the same meaning. A
further analysis of vocabulary in test cases is our future work.
The cut level, a parameter of test case clustering, plays animportant role in our proposal. As mentioned above, anothercut level can produce another performance in recommending
test cases. While we set 0.3 as the cut level based on
experience, our experience might not work well for anotherdata set: if we use another set of test cases which are preparedby other engineers of another organization, we would have toreexamine the cut level. We plan to perform a further analysis
of cut level and seek a better way of deciding an appropriate
cut level in the future.
V. C ONCLUSION
We focused on the content of test cases, and evaluated thesimilarity between test cases through a morphological analysis.
Based on those similarities, we categorized test cases into clusters. Then, we proposed a cluster-based method with which torecommend test cases for the regression testing—once a test
case is selected for a regression testing, the proposed method
recommends other test cases which should also be run inorder to reduce the risk of overlooking regressions (failures).
The proposed method also leveraged the Mahalanobis-Taguchi
method for prioritizing test cases within the recommendation
set of test cases.
We conducted an empirical study with a real product, andthe results showed that the proposed method successfully
recommended test cases which found latent regressions. The
Mahalanobis-Taguchi method also worked well for prioritizing
test cases. Moreover, we saw the following finding: if moretest cases are recommended by the proposed method at aversion, there may be a higher risk of overlooking regressions
at that version. Thus, the proposed method would be helpfulin evaluating a test plan as well.
Our future work includes the following: (1) to apply the proposed method to other products for the purpose of examining
its generality; (2) to perform further analyses on the parameterof the clustering algorithm and on the features of test cases
from the perspective of the natural language analysis.
R EFERENCES[1] T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel,“An empirical study of regression test selection techniques,” ACM
Transactions on Software Engineering and Methodology, vol. 10, no. 2,
pp. 184–208, Apr. 2001.
[2] S. Yoo and M. Harman, “Regression testing minimization, selection andprioritization: A survey,” Software Testing, Verification and Reliability,
vol. 22, no. 2, pp. 67–120, Mar. 2012.
[3] D. Jeffrey and N. Gupta, “Test case prioritization using relevant slices,”in Proc. 30th Annual International Computer Software and Applications
Conference, vol. 1, 2006, pp. 411–420.
[4] S. Mirarab and L. Tahvildari, “A prioritization approach for software
test cases based on bayesian networks,” in Proc. 10th International
Conference on Fundamental Approaches to Software Engineering, 2007,
pp. 276–290.

[5] J.-M. Kim and A. Porter, “A history-based test prioritization techniquefor regression testing in resource constrained environments,” in Proc.
24th International Conference on Software Engineering, 2002, pp. 119–129.
[6] H. Aman, M. Sasaki, K. Kureishi, and H. Ogasawara, “Application of the
0-1 programming model for cost-effective regression test,” in Proc. 37th
Annual International Computer Software and Applications Conference,
July 2013.
[7] H. Aman, Y. Tanaka, T. Nakano, H. Ogasawara, and M. Kawahara, “Application of Mahalanobis-Taguchi method and 0-1 programming method
to cost-effective regression testing,” in Proc. 2016 42nd Euromicro
Conference on Software Engineering and Advanced Applications, Aug.
2-16, pp. 240–244.
[8] M. Sherriff, M. Lake, and L. Williams, “Prioritization of regression
tests using singular value decomposition with empirical change records,”
in Proc. 18th IEEE International Symposium on Software Reliability
Engineering, 2007, pp. 81–90.
[9] R. Carlson, H. Do, and A. Denton, “A clustering approach to improving
test case prioritization: An industrial case study,” in Proc. 27th IEEE
International Conference on Software Maintenance, 2011, pp. 382–391.
[10] D. Leon and A. Podgurski, “A comparison of coverage-based anddistribution-based techniques for filtering and prioritizing test cases,”in Proc. 14th International Symposium on Software Reliability Engineering, 2003, pp. 442–453.
[11] M. J. Arafeen and H. Do, “Test case prioritization using requirementsbased clustering,” in Proc. IEEE 6th International Conference on Software Testing, Verification and Validation, 2013, pp. 312–321.
[12] Y. Ledru, A. Petrenko, S. Boroday, and N. Mandran, “Prioritizing test
cases with string distances,” Automated Software Engineering, vol. 19,
no. 1, pp. 65–95, Mar. 2012.
[13] S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein, “Static test
case prioritization using topic models,” Empirical Software Engineering,
vol. 19, no. 1, pp. 182–212, Feb. 2014.
[14] G. Salton, Automatic text processing: the transformation, analysis, andretrieval of information by computer. Boston, MA: Addison-WesleyLongman Publishing, 1989.
[15] G. Taguchi, S. Chowdhury, and Y. Wu, The Mahalanobis-Taguchi
System. New York, NY: Mc, 2001.