A Topic Model and Test History-Based Test Case
Recommendation Method for Regression Testing
Hirohisa Aman∗ , Takashi Nakano† , Hideto Ogasawara† and Minoru Kawahara∗∗ Center for Information Technology, Ehime University, Matsuyama, Ehime 790-8577, Japan† Corporate Software Engineering & Technology Center, Toshiba Corporation, Kawasaki, Kanagawa 212–8585, Japan

Abstract—In the regression testing, an oversight of a regression
is a serious problem to be avoided. A test engineer usually selects
test cases to rerun for a regression testing. While the selection isa useful expert decision, there is also a risk of missing some
important test cases. To support a more effective regression
testing, this paper focuses on the following two kinds of data:
1) the similarity of test cases in terms of their topics, and 2)
the test history. Then, the paper proposes a hybrid method forrecommending test cases in two steps by using the above data. As
the thirst step, it recommends test cases which are highly-similar
to the manually-selected ones. Then, as the second step, themethod recommends the remaining test cases in decreasing orderof priority computed by using the test history. The usefulness ofthe proposed method is proved through an empirical study using
a set of regression test data for an industrial product.

I. I NTRODUCTION
A software product usually evolves through program modifications which aim at enhancing functionalities or fixing faultsof the product, and such a software evolution is required toprovide an improved service to the users. However, thosemodifications may also have risks of creating other faults(regressions) [1]. In order to avoid releasing a faulty versionof the product, a careful regression testing—rerunning test
cases—should be performed when the product is changed.
To detect all regressions perfectly, it is ideal to rerun all
test cases whenever a part of the target product is changed.
However, rerunning all test cases would be unrealistic because
it requires a lot of time and effort [2] but it may be inefficient
to detect regressions: most of all test cases pass almost
always, and only a few ones may fail fairly infrequently.
Therefore, there have been many studies to prioritize test cases
toward a more efficient regression testing [3]. The test caseprioritization studies are classified into three types according
to what they are mainly based on: (1) the source programs to
be tested, (2) the test history and (3) the content of test cases.
The first type of study is to prioritize test cases by analyzing
the source programs to be tested. The common way among
these studies is to make links between a test case and its
corresponding programs which are tested by the test case.
When a source program is modified, those studies single out
the corresponding test cases which are associated with themodified parts and/or evaluate a test case’s degree of closenessto the modified programs (e.g., [4], [5]).
The second type of study utilizes the test history which
consists of the testing results such as the version of tested
product, the IDs of used test cases and their results (passes

or fails). This type of study evaluates a test case’s priority byusing its failure detection rate in the past and/or its version
gap between the current version and the last-run one [6], [7].
While the test history-based methods are not directly related toprogram modifications, they can be useful to prevent overlooking unexpected regressions which are apparently independentof the modified programs.
The third type of study focuses on the content of test cases.
Test cases are either test programs (for an automated test)
or test scripts (for a manual test). In this type of study, test
cases are analyzed by using the natural language processing
techniques such as the morphological analysis and the topic
analysis, and their similarities or dissimilarities to each otherare evaluated. Then, test cases are prioritized for a better
regression testing, by using their similarities/dissimilarities tothe set of already selected (being rerun) test cases [8], [9].
When a test engineer performs a regression testing for a
product, he/she does not always know anything about thecorresponding code modifications made to the product [10].
Furthermore, if the coding work is an (offshore) outsourcing,
it would be harder to obtain the details of modifications. In
such a case, we cannot utilize the above first type of study
which is based on the source program analysis. Since our (theauthors’) context is in this case, we will focus on the second
and the third types of study—using the test history and thecontent of test cases—in this paper.
For a regression testing, test engineers may select test cases
to rerun from their test case pool in accordance with theirknowledge and/or experience. We believe those selections areuseful because it is a kind of expert decision. However, there
would also be a risk of overlooking regressions. Hence, while
respecting their first selection, we consider recommending
additional test cases which are useful to detect regressions. The
key contribution of this paper is to propose a novel method
for making a useful recommendation to prevent overlooking
regressions by utilizing the test history and the topic analysis
to the test cases [11], which is a combination of the above
second and third types of study.
The remainder of this paper is organized as follows. Section
II explains the test case recommendation in our context andrelated previous work, then proposes a novel method. Section
III reports the empirical study that we conducted to show ausefulness of our proposal. Section IV briefly describes therelated work. Finally, Section V presents our conclusion andfuture work.

II. E FFECTIVE T EST C ASE R ECOMMENDATION FOR
R EGRESSION T ESTING
A. Test Case Recommendation Problem
We first explain our context of regression testing. For aregression testing of a product, we have a set of test cases(test case pool). These test cases are written in Japanese andthey are test scripts to manually operate the product by atest engineer. At a regression testing, the test engineer knows
which functionalities were upgraded and/or which bugs were
fixed. Then, the test engineer singles out some test cases fromthe pool, which seem to be related to the modifications.
Now, in order to reduce a risk of overlooking regressions,
we want to recommend adding other test cases which areremained in the pool and yet seem to be useful to detect regressions (see Fig. 1). Since the testing is performed manually,
the recommendation set should be minimal because of their
testing cost. Our available data are the test history (see TableI) and the content of test cases (see Fig. 2).
Table I shows an example of test history. In the table,
symbols “P,” “F” and “–” signify the “pass,” “fail” and “norun” of the corresponding test case at the corresponding
version, respectively. For example, test case t2 passed atversion v2 and failed at v4 , and t3 was not run at v1 . In
Table I, v5 is the version at which we are about to perform aregression testing. Symbol “✓” means that the corresponding
test cases are selected by the test engineer; the remaining test
cases shown as “?” are the candidates of our recommendation.
Our goal is to recommend useful test cases from the candidate set. A better recommendation means that more regressions
are detected by fewer test cases, i.e., test cases with higher
priorities are more likely to detect regressions. This is formallydefined as the following test case recommendation problem.
Definition 1 (The test case recommendation problem):Given the set of test cases, T , which consists of n test cases:
T = {ti }ni=1 . Suppose a test engineer selected m test cases
t1,j from T (for j = 1, . . . , m); Let H = {t1,j |t1,j ∈ T }m
j=1 .
Then, recommend the remaining test cases in decreasing orderof ∧priority, as a sequence (t2,r )r=1,...,n−m where (t2,r ∈T t2,r ̸∈ H) and t2,r has a higher priority than t2,r′ if r < r′(for r = 1, . . . , n − m).

TABLE IE XAMPLE OF TEST HISTORY.
test
case
t1
t2
t3
t4
t5
..
.

Hover the mouse cursor over the “?” icon. Then, check if the corresponding help message is displayed in a pop-up window.

Fig. 2. Example of test case (translated into English).

v2
–
P
P
F
–
..
.

version
v3
v4
–
–
P
F
–
–
P
P
–
–
..
..
.
.

v5
?
✓
?
?
✓
..
.

(P: pass; F: fail; –: no run; ✓: selected; ?: candidate)

The usefulness of recommendation is evaluated by the area
under the curve of the Alberg diagram (AUCA)1 [12]. A higher
AUCA value means a better recommendation because moretest cases succeeded in detecting regressions, in an earlier
phase of the regression testing.
□Figure 3 shows examples of Alberg diagrams. The best caseis in Fig. 3 (a) since all regressions are detected by the higher
priority test cases. If some high-priority test cases fail to detect
regressions, the diagram becomes like Fig. 3 (b) and its AUCA
value (the area under the curve) gets smaller than (a).
B. Test Case Recomendations Using Test History and/or Content of Test Cases
To recommend (prioritize) test cases, we use the test history
and the content of test cases. In this subsection, we describe
methods proposed in the previous work.
In the test history-based prioritization, the following two
types of data are mainly focused:[Type-1] Whether a test case was run or not at a version.
[Type-2] Whether a test case failed or not at a version.
1) Application of the Exponential Weighted Moving Averageand the Exponential Smoothing: Kim and Porter [6] proposed
to leverage the exponential weighted moving average and theexponential smoothing [13]. For a pair of test case ti andversion vj , they defined the selection probability of ti as:
P (ti , vj ) = α · hij + (1 − α)P (ti , vj−1 ) ,(1)where hij ∈ {0, 1} is the test history observation for ti at
vj , and α is a smoothing constant used to weight individual
history observations (0 ≤ α ≤ 1); P (ti , v0 ) = hi1 .

(a) the best caseFig. 1. Outline of test case recommendation.

v1
P
F
–
–
P
..
.

(b) another caseFig. 3. Examples of Alberg diagrams.
1 The vertical axis signifies the accumulated number of faults (regressions)in the original Alberg diagram. However, we replace it by the accumulated
number of regression-detected test cases because we cannot trace the faults
in the source code; While the test cases and the test history are available, thesource code is not available in our context.

A higher P (ti , vj ) means a higher priority of ti for aregression testing at the next version, vj+1 . Kim and Porter
proposed to use Eq. (1) for both the above two types of data.
For type-1 data, let hij = 1 if ti was “not run” at vj ;otherwise, hij = 0. Then, they used Eq. (1) with a low α,e.g., α = 0.1. If ti has not been run for more versions until
vj , P (ti , vj ) gets larger. For example, let us consider t1 shownin Table I. Here, we have (h1k ) = (0, 1, 1, 1). When α = 0.1,
their selection probabilities are upgraded from v1 to v4 as:
P (t1 , vk ) = 0 → 0.1 → 0.19 → 0.271 .
For type-2 data, let hij = 1 if ti “failed” at vj ; otherwise,
hij = 0. In this case, they proposed to use Eq. (1) with ahigh α, e.g., α = 0.9. If ti has consecutively failed for moreversions until vj , P (ti , vj ) gets larger. Let’s take t2 shownin Table I for example. We have (h2k ) = (1, 0, 0, 1). Whenα = 0.9, their selection probabilities change from v1 to v4 as:
P (t2 , vk ) = 0.9 → 0.09 → 0.009 → 0.9009 .
For the sake of convenience, we will call the above methods
using the selection probabilities for type-1 and type-2 as“SP 1” and “SP 2,” respectively, in the following sections.
2) Application of the Mahalanobis-Taguchi Method: Amanet al. [7] proposed to integrate the above two types of datausing the notion of the Mahalanobis-Taguchi method2 [14].
In [7], Type-1 and type-2 data are measured by the following
two metrics GLC and FR, respectively.
1) The Gap between the Last run version and the Current
version (GLC): For a test case ti , GLC(ti ) is the numberof consecutive versions at which ti was not run until thecurrent version.
2) The Failure Rate (FR): For a test case ti , FR(ti ) is thefailure rate of ti until the current version. That is to say,
it is the number of versions failed by ti , divided by the
number of versions at which ti was run.
For example, GLC(t1 ) = 4, GLC(t4 ) = 1, FR(t1 ) = 0 andFR(t2 ) = 2/4 in Table I, where the current version is v5 .
A test case having a greater GLC value has not been run
for more versions after its last run. Therefore, such a test case
may have a higher risk of overlooking regressions. While test
engineers often select test cases which seem to be related withthe functionalities upgraded at the version, unexpected parts
might include regressions. It is better to rerun other test cases
as well, even if they seem to have no relation with the changes.
A test case having a higher FR value has a better trackrecord for finding a regression in the past. Such a test case
may test a part which is likely to be faulty in the product, andwe can expect a higher ability to find a regression again. It isbetter to rerun as many such test cases as possible.
On both of these metrics, value 0 represents the corresponding test case has the lowest expectation to find a regression.
Thus, the distance between xi T = (GLC(ti ), FR(ti )) and
0T = (0, 0) can express the worth of ti to rerun in termsof these metrics. A straightforward way of computing the
2 The Mahalanobis-Taguchi method is one of statistical quality control
methods, which leverages the Mahalanobis distance to detect abnormal (poorquality) objects, etc. See [14] for the details.

√(xi − 0)T (xi − 0) , which is the
distance is dE (xi ) =Euclidean distance. However, it has a lack of consideration forthe dispersion of data. Since metric GLC has a wider range
than FR, GLC can have a greater impact on the computed
distance. The following Mahalanobis distance is a better
choice to consistently integrate the effects of two metrics:√(xi − 0)T S −1 (xi − 0) ,(2)
dM (xi ) =where S is the covariance matrix [15] of GLC values andFR values, and S −1 is its inverse matrix. Since it uses thecovariance matrix, the Mahalanobis distance can consider not
only the dispersion on each axis (a certain metric) but also
their covariances of data, i.e., correlations between metrics.
For example, if we have the following covariance matrix S,[][]
4.000 −0.200
0.3125 1.2500
−1
S=
, S =
,
−0.200 0.050
1.2500 25.000
we obtain dM (x1 ) ≃ 2.24, dM (x3 ) ≃ 1.68 and dM (x4 ) ≃1.98. On the other hand, by using the Euclidean distance, we
get dE (x1 ) = 4, dE (x3 ) = 3 and dE (x4 ) ≃ 1.05. That
is to say, although t1 is the worthiest in {t1 , t3 , t4 } for boththe Euclidean distance and the Mahalanobis one, the second
worthiest test cases are different between two distances; In the
Euclidean distance, the value of GLC seems to be dominant.
For the sake of convenience, we hereinafter call the above
method using the Mahalanobis distance as “M D.”
3) Content-Based Clustering of Test Cases and Combination with MD: We have focused on the content of test casesin the past [9]. In [9], we proposed to recommend test casesin the following two steps:
1) First, categorize the test cases into some clusters in termsof their similarities, based on their mutual words.
2) Then, recommend test cases from the same cluster as
the manually selected test cases3 . The recommendation
is made by the M D method mentioned above.
When an engineer selected a test case tx and there is similar
but not identical another test case ty , the engineer maybe
should run not only tx but also ty . For example, suppose wewant to test two different functions, A and B, under a certain
condition: tx executes in the order (A,B), and ty does in the
opposite order (B,A). In such a case, the contents of tx andty must be similar. However, since they are not identical, we
should not skip over either tx or ty ; we have to rerun both ofthem. Some test engineers might rerun only tx , and overlook
a regression which would have been detected by ty .
While our previous method [9] partially succeeded to recommend test cases which can detect regressions, it had thefollowing challenge: how to deal with the variation of words
among test cases. This is from a linguistic feature in ourdataset. Our dataset was the set of test cases written in anatural language, Japanese. Hence, different people may write
different phrases with using different words for the same orsimilar purpose. Especially, Japanese tends to have a high
3 In the case of Table I, the manually selected test cases are t and t .
2
5

vagueness of phrase. Since the previous work considered thesimilarity between test cases by using the exact match ofmutual words, such a vagueness may be a serious threat tovalidity. To solve the above challenge in our previous work, wewill propose to apply the topic model in the next subsection.
C. Proposal
The topic model is a popular model for analyzing documents written in a natural language [16], [17]. In the modeling, the words are extracted from the documents and theirco-occurrence relationships are analyzed. Through the cooccurrence analysis, each word is liked to one or more topics,
where topics are latent variables (see Fig. 4). This liking work
is like a clustering of words, but it does not limit word’s cluster
to a specific one. Finally, each document is associated to one
or more topics. While topics do not explicitly appear in adocument, the document’s feature is expressed by the feature
vector (p1 , . . . , pk ) where pi denotes the probability that the
document has the i-th topic; k is the number of topics. The
topic modeling is performed by the latent Dirichlet allocation(LDA) [18] which is a popular statistical method to estimate
the latent topics in documents. There are implementationsof the LDA for natural language documents: for example,
topicmodels package4 is available for R5 .
By using the topic model, we can solve the challengementioned in Section II-B3, i.e., vagueness of words. We
now can consider the similarity between test cases by not themutual words but the latent semantics of words. That is to say,since the feature vector (p1 , . . . , pk ) presents the membership
degrees to the topics, the semantic similarity between test cases
can be computed by using the feature vectors. We define thesimilarity between test cases tx and ty as
1 ∑|pxi − pyi | ,k i=1ksim(tx , ty ) = 1 −(3)where k is the number of topics, and pxi and pyi are theprobabilities that tx and ty have the i-th topic, respectively. It
is based on the Manhattan distance, and the reason why weuse it is from the experimental results by Thomas et al. [8].
Now we propose a test case recommendation method as thefollowing two steps:

Fig. 4. Image of latent topics.

[Step-1] Suppose m test cases { t1,i }i=1,...,m are manually
selected by a test engineer. Define the similarity ofa remaining test case tx to the selected set ass(tx ) =min [ sim(tx , t1,i ) ] .

i=1,...,mThen, recommend test cases having high similarities with the manually-selected set.
[Step-2] Recommend the remaining test cases by the SP 1,
SP 2 or M D method.
The first step uses only the similarity of test case. If ahighly-similar test case is remained (not selected), it may cause
an overlooking of regression. We will decide an appropriate
threshold of similarity in our empirical study (Section III).
In the second step, we follow the conventional prioritization
method (SP 1, SP 2 or M D). The above proposal is the
key contribution of this paper, which combines the previous
methods mentioned in Section II-B: Once a manually-selected“seed” subset of test cases is given, the proposed method
automatically prioritize the remaining test cases in the pool.
III. E MPIRICAL S TUDY
A. Dataset
To examine our proposal, we obtained data of regression
testing for a Web-based system which has been developed andmaintained by Toshiba corporation6 . We have 300 test cases(n = 300) and a test history for 13 versions {vj }13
j=1 of the
product. For this study, we manually reran all of 300 test cases
at the latest version. It took 539 minutes and revealed that
there are regressions overlooked by a past regression testing—
22 test cases failed at the latest version. The oversight was
occurred at version v7 . For v7 , seven (m = 7) test cases weremanually selected from the test case pool, but more test cases
should be selected to detect regressions earlier.
In this section, we examine if the proposed method makes abetter recommendation for the regression testing at v7 , where
we consider v7 to be the current version for computing metricvalues mentioned above.
B. Procedure
Our empirical study is conducted in the following steps.
1) Build a topic model of our test cases. Prior to the
topic modeling, we have to extract words from the test
cases. Since they are written in Japanese, we perform atokenization, a stemming, an elimination of stop words,
a morphological analysis, and an extraction of words7 by
MeCab8 which is a popular analyzer for Japanese.
For a model construction, the number of topics (k) mustbe given as a key parameter.
2) Given manually-selected test cases {t1,j }j=1,...,m . For
each test case tx remained in the test case pool, computes(tx ) according to Eq.(4).

4 https://cran.r-project.org/web/packages/topicmodels/

6 We omit the details of the product since they are confidential data.

5 R is one of the popular free software environments for statistical computing

7 Nouns, verbs and adjectives are extracted in this study.

and graphics. https://www.r-project.org/(4)

8 http://taku910.github.io/mecab/

Our empirical results showed that the proposed method
using τ = τ1 or τ2 is the best, i.e., it is useful to rerun themost 1% or 2% similar test cases at first.
TABLE IIE MPIRICAL RESULTS (AUCA VALUES )history only
τ1
τ2
τ3
τ4
τ5

SP 1
5166
5409 (+4.7%)
5374 (+4.0%)5320 (+3.0%)5265 (+1.9%)
5211 (+0.9%)

history-based method
SP 2
MD
4252
5318
4339 (+2.0%) 5386 (+1.3%)
4390 (+3.2%)5380 (+1.2%)
4333 (+1.9%)5325 (+0.1%)4295 (+1.0%)5269 (−0.9%)4238 (−0.3%)5212 (−2.0%)

9 To decide the number of topics, we first tried using ldatuning packageof R, https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html,
which provides topic evaluations using four metrics. Unfortunately, thesefour metrics showed different tendencies and it was hard to decide a specificnumber of topics. Therefore, we dare to use the principal component analysis
as a substitute way of determining the number of topics while it is notorthodox. We need a further study for a better way in the future.

While our recommendation of highly-similar test cases
worked for a successful detection of regressions in this empirical study, it might depend on the product’s features, theway of testing, or even who did the regression test. To provea generality of our results, we also need to perform further
experiments using other products in the future.
Since we could not trace a link between a code change anda test case in our context, we have discussed the efficiencyof testing with the number of “failed test cases” but not the
number of “faults.” That is to say, two or more test cases
might detect the same fault. This is our big threat to validity
and limitation in this study. In order to analyze the impact ofsuch a difference of viewpoint, we have to do further analyses
using more detailed testing and fault data in the future.
We used the LDA for our topic modeling. While the LDA
is the most popular method, the performance depends on theparameters used for the computation. We adopted the default
parameters of the lda function included in topicmodels
package. Although the default parameters might not be appropriate to the test cases written in Japanese, we did not tune the

Proposed

Accumulated # ofregression−detected test cases
5
10
15
20
25

D. Discussions

E. Threats to Validity

Conventional

Proposed

Conventional0

To decide the number of topics (k), we first built a topic
model with many (= 100) topics and performed the principal
component analysis9 . Because the top 19 principal components
can explain over 90% information of the original space, we
decided the number of topics to be 19 in this empirical study.
We also have to decide the threshold of similarity (τ ). Since
we have no theoretical basis to decide a proper τ , we will try
some values: τx (for x = 1, 2, 3, 4, 5) where τx corresponds
to the top x% in the similarity distribution.
Table II shows the AUCA values of recommendations. In
the table, the row labeled as “history only” presents the AUCA
values of the conventional history-based recommendations:
SP 1 (α = 0.1), SP 2 (α = 0.9) and M D. Each row of τx(for x = 1, . . . , 5) gives the AUCA values of recommendations
made by the proposed method using τ = τx and the corresponding column’s history-based method (SP 1, SP 2 or M D).
A percentage shown in parentheses denotes the differenceof the corresponding AUCA values between the proposed
method and the conventional (history only) one: for example,
the AUCA value of the recommendation made by using only
SP 2 is 4252, and the AUCA value of the one made by the
proposed method using τ1 and SP 2 is 4339, so the percentage
of difference is obtained as: (4339 − 4252)/4252 ≃ 2.0%.
For each column in Table II, the highest AUCA value isindicated by boldface. Regardless of the used history-based
method, the proposed method with τ = τ1 or τ2 show thebest performance in recommending test cases in our dataset.

Accumulated # ofregression−detected test cases
5
10
15
20
25

C. Results

Because of their high similarities, the test engineer might
omit them as unimportant ones: Suppose there is a highlysimilar pair of test cases, tx and ty . When the engineer rerantx and it passed, he/she would highly expect that ty also passes
since the highly-similar tx passed. Then, he/she might consider
that it does not matter to omit a rerunning of ty . However,the AUCA value gets higher by prioritizing highly-similar test
cases first, so the results prove an importance of focusing on
similar test cases. Figure 5 (a) presents the Alberg diagramsof the conventional method and the proposed one using τ1with M D (whose X-axis is limited to 100). At the kickoff
point in Fig. 5 (a), highly-similar test cases successfully detect
regressions which were overlooked by manually-selected ones.
On the other hand, the usefulness of recommending similar
test cases seems to be limited to highly-similar ones. Indeed, τ5
showed the lowest AUCA values for all methods (see Table II).
Figure 5 (b) presents the Alberg diagrams of the conventional
method and the proposed one using τ5 with M D. Althoughhighly-similar test cases are really useful (see Fig. 5(a)),
recommending more test cases seems to delay the growth ofregression detections (see Fig. 5(b)). Since a regression testingwith only similar test cases may have a lower coverage, sucha delay of growth in detecting regressions would be caused.

03) Recommend highly-similar test cases {t2,r |s(t2,r ) ≥ τ }in decreasing order of similarity, where τ is the thresholdof similarity; we will empirically decide τ .
4) Recommend the remaining test cases by one of the conventional history-based methods—SP 1, SP 2 or M D.

0

20

40
60
Recommendation(a) τ = τ1

80

100

0

20

40
60
Recommendation(b) τ = τ5

Fig. 5. Comparisons of Alberg diagrams.

80

100

parameters because the tuning can be another threat to validity.
A further analysis on the parameter tuning is our future work.
IV. R ELATED W ORK
Thomas et al. [8] utilized the topic model to prioritize
test cases. In their method, dissimilarities between test cases(programs) are computed by using the Manhattan distanceof corresponding topic vectors. They proposed to iteratively
select test cases in decreasing order of dissimilarity to thealready-selected test cases in order to expand the test coverage. Hemmati et al. [19] applied the notion of the topicbased prioritization proposed in [8] to test cases written in anatural language: they prioritized test cases by maximizing
dissimilarities to already-prioritized test cases in terms of
their topics. Our fundamental way of evaluating the similarity/dissimilarity between test cases is common to [8], [19].
While these previous work prioritized test cases to get a higher
diversity with a fewer test cases, we have another viewpoint—
we dare to recommend test cases which are highly-similar
to manually-selected ones. The aim of such an opposite wayof prioritization is to avoid a human error. Since such ourprioritization would produce a lower coverage, we have ahybrid strategy: after we recommend a few highly-similar test
cases, we switch our strategy to the history-based method.
Saha et al. [20] proposed to reduce a test case prioritization
problem to an information retrieval one. In their approach,a code change is corresponding to a query, and test cases
are regarded as a set of documents. Once a code change is
given, test cases are prioritized in terms of the similarity to
the change, where the similarity is computed by the tf/idf
term weighted vector space model [21]. They reported thatthe proposed method works better to effectively prioritize test
cases through an empirical study using open source projects,
and the usefulness of leveraging a natural language processingbased similarity evaluation for prioritizing test cases is proved.
Although we cannot get the code change information in ourcontext, it is common that test cases are prioritized based on
the similarity to a seed data—while Saha et al. focused on acode change, we used a manually-selected test cases.
V. C ONCLUSION
To enhance the manual regression testing, we focused on
the similarity between test cases using the topic model, andproposed a novel method for recommending test cases. Whensome test cases are manually selected for a regression testing,the proposed method recommends additional test cases in thefollowing two steps. As the first step, the method recommends
test cases which are highly-similar to the manually-selected
ones. Then, as the second step, the method recommends test
cases in decreasing order of priorities which are computed by
the conventional test history-based method. Through an empirical study, we proved that the proposed method successfullyenhances the performance of regression testing. Notice that thefirst-step recommendation is limited to “highly” similar test
cases only in order to prevent a human error in the manual
selection; if we simply prioritized all test cases in decreasingorder of similarity, it would not effectively work to detectvarious regressions. The proposed method uses both the test
case similarity and the test history in a hybrid manner.
Our future work includes: 1) a further study to automaticallydecide a better threshold of the similarity between test cases,
and 2) a further analysis using other test data of other products
to examine the general usefulness of the proposed method.
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers
for their helpful comments on an earlier version of this paper.
R EFERENCES[1] C. Jones, Applied Software Measurement: Global Analysis of Productivity and Quality, 3rd ed. New York, McGraw-Hill, 2008.
[2] J. Hartmann and D. J. Robson, “Techniques for selective revalidation,”
IEEE Software, vol. 7, no. 1, pp. 31–36, Jan. 1990.
[3] S. Yoo and M. Harman, “Regression testing minimisation, selection andprioritisation: A survey,” Softw. Testing, Verification & Rel., vol. 22,
no. 2, pp. 67–120, Mar. 2010.
[4] D. Jeffrey and N. Gupta, “Test case prioritization using relevant slices,”in Proc. 30th Annual Int’l Comp. Softw. & App. Conf., vol. 1, Sept.
2006, pp. 411–420.
[5] S. Mirarab and L. Tahvildari, “A prioritization approach for software
test cases based on bayesian networks,” in Proc. 10th Int’l Conf.
Fundamental Approaches to Softw. Eng., Mar. 2007, pp. 276–290.
[6] J.-M. Kim and A. Porter, “A history-based test prioritization techniquefor regression testing in resource constrained environments,” in Proc.
24th Int’l Conf. Softw. Eng., May 2002, pp. 119–129.
[7] H. Aman, Y. Tanaka, T. Nakano, H. Ogasawara, and M. Kawahara,“Application of Mahalanobis-Taguchi method and 0-1 programming
method to cost-effective regression testing,” in Proc. 42nd EuromicroConf. Softw. Eng. & Advanced App., Aug. 2016, pp. 240–244.
[8] S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein, “Static
test case prioritization using topic models,” Empir. Softw. Eng., vol. 19,
no. 1, pp. 182–212, Feb. 2014.
[9] H. Aman, T. Nakano, H. Ogasawara, and M. Kawahara, “A test caserecommendation method based on morphological analysis, clusteringand the Mahalanobis-Taguchi method,” in Proc. 10th IEEE Int’l Conf.
Softw. Testing, V. & V. Workshops, Mar. 2017, pp. 29–35.
[10] M. J. Arafeen and H. Do, “Test case prioritization using requirementsbased clustering,” in Proc. 6th Int’l Conf. Softw. Testing, V. & V., Mar.
2013, pp. 312–321.
[11] T. K. Landauer, D. S. McNamara, S. Dennis and W. Kintsch, Ed.,
Handbook of Latent Semantic Analysis. London, Routledge, 2014.
[12] N. Ohlsson and H. Alberg, “Predicting fault-prone software modules
in telephone switches,” IEEE Trans. Softw. Eng., vol. 22, no. 12, pp.
886–894, Dec. 1996.
[13] R. G. Brown, Statistical Forecasting for Inventory Control. New York,
McGraw-Hill, 1959.
[14] G. Taguchi, S. Chowdhury, and Y. Wu, The Mahalanobis-Taguchi
System. New York, McGraw-Hill, 2001.
[15] G. J. G. Upton and I. Cook, A Dictionary of Statistics, 2nd ed. Oxford
University Press, 2008.
[16] T. Hofmann, “Probabilistic latent semantic indexing,” in Proc. the 22ndInt’l ACM SIGIR Conf. Research & Dev. in IR, Aug. 1999, pp. 50–57.
[17] D. M. Blei, “Probabilistic topic models,” Commun. ACM, vol. 55,
no. 4, pp. 77–84, Apr. 2012.
[18] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” J.
Machine Learning Research, vol. 3, pp. 993–1022, Jan. 2003.
[19] H. Hemmati, Z. Fang, M. V. Mäntylä, and B. Adams, “Prioritizing manual test cases in rapid release environments,” Softw. Testing, Verification& Rel., vol. 27, no. 6, pp. e1609:1–25, Sept. 2017.
[20] R. K. Saha, L. Zhang, S. Khurshid, and D. E. Perry, “An information
retrieval approach for regression test prioritization based on programchanges,” in Proc. 37th Int’l Conf. Softw. Eng., May 2015, pp. 268–
279.
[21] J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed?–more
accurate information retrieval-based bug localization based on bug
reports,” in Proc. 34th Int’l Conf. Softw. Eng., June 2012, pp. 14–24.