Selecting Test Cases based on Similarity ofRuntime Information:
A Case Study of an Industrial Simulator
Kazumasa Shimari∗ , Masahiro Tanaka† , Takashi Ishio∗ , Makoto Matsushita† , Katsuro Inoue‡ , Satoru Takanezawa§∗ Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan

Email: {k.shimari, ishio}@is.naist.jp† Graduate School of Information Science and Technology, Osaka University, Osaka, Japan

Email: {m-tanaka, matusita}@ist.osaka-u.ac.jp
Email: inoue599@nanzan-u.ac.jp§ Daikin Industries, Ltd., Osaka, Japan. Email: satoru.takanezawa@daikin.co.jp‡ Faculty of Science and Technology, Nanzan University, Aichi, Japan.

Abstract—Regression testing is required to check the changes
in behavior whenever developers make any changes to a softwaresystem. The cost of regression testing is a major problem because
developers have to frequently update dependent components
to minimize security risks and potential bugs. In this paper,
we report a current practice in a company that maintains an
industrial simulator as a critical component of their business. The
simulator automatically records all the users’ requests and thesimulation results in storage. The feature provides a huge numberof test cases for regression testing to developers; however, theirtime budget for testing is limited (i.e., at most one night). Hence,the developers need to select a small number of test cases to
conﬁrm both the simulation result and execution performance
are unaffected by an update of a dependent component. In
other words, the test cases should achieve high coverage while
keeping diversity of execution time. To solve the problem, we
have developed a clustering-based method to select test cases,
using the similarity of execution traces produced by them. The
developers have used the method for a half year; they recognizethat the method is better than the previous rule-based method
used in the company.
Index Terms—Test Selection, Dynamic Analysis, Software Dependency, Clustering

I. I NTRODUCTION
Regression testing is performed to check the changes inbehavior before and after a software change. The cost ofregression testing is a signiﬁcant problem for software developers because they have to repeatedly perform regression
testing even when the software is not directly changed.
Daikin Industries, Ltd., a collaborating company, developed
an industrial thermodynamic and ﬂuid mechanic simulator.
The users of this simulator are engineers of the company who
design new products. The simulator accepts various parametersspecifying a structure of an electromechanical product andsimulates the physical behavior; the users can estimate the
efﬁciency of the structure without constructing an actual
prototype. The simulator is now recognized as one of the mostimportant simulators for their product development.

Software developers in the company periodically update thesimulator and its dependencies and then perform regression
testing. For effective regression testing, the simulator automatically records all the users’ requests and the simulation resultsin storage. When the developers update the simulator, they
execute some simulations again and compare the new resultswith the recorded results. Instead of all the recorded requests,
the software developers selected a small subset of test casesthat cover available simulation components so that they canﬁnish their test within a limited time budget.
Although the software developers believed their strategy
was sufﬁcient, they failed to detect an incompatibility between
JDK versions. They updated the JDK version from Oracle JDK
8 to AdoptOpenJDK 11 and performed regression testing as
usual. However, a simulator failure occurred a few months
after the regression testing. The cause was a change in the
behavior of the Java standard library, which prevented several
simulations from running properly. Even though the test cases
have been selected to cover simulation components, they did
not cover a corner case in a wide variety of behavior of
the simulation components. Although test case selection is a
popular topic in the software testing community, we could not
identify a suitable method to select a small number of effective
test cases that can be executed within at most one night froma large number of test cases.
In this paper, we report our current regression testing
method developed for this simulator. The objective of themethod is to select test cases satisfying the following concernsof software developers in the company:• The time budget for regression testing is limited. They
would like to select only a small number of test cases.
• The selected test cases should exercise functional components as much as possible. In other words, code coverage
should be high.
• The selected test cases should include both short simulations and long simulations reﬂecting regular usage
patterns so that they can check the performance of thesimulator after an update.
To address the concerns, we developed a method to select test
cases using their runtime information. Our method employs
k-means clustering to identify similar executions and selectrepresentative test cases from each of the clusters. As theparameter k corresponds to the number of test cases, developers can easily specify a value according to their time budget.
The developers have used the method to dependency updatetasks for a half year; they successfully update dependent
components.
In the remainder of the paper, Section II describes our test
case selection method. Then, Section III describes the casestudy, and Section IV applies our method to an industrial simulator. Section V describes the related work. Finally, Section VI
concludes the paper and describes future work.
II. O UR T EST C ASE S ELECTION M ETHOD
Our method takes as input a set of test cases and a parameterk that speciﬁes the number of test cases to be selected. Our
method consists of two steps. Figure 1 shows an overview
of our method. In STEP Ⅰ, we collect runtime informationfor each test case by executing test cases with an execution
trace recorder. This step is performed in advance. In STEP Ⅱ,
we classify test cases based on the similarity of their runtime
information and select their representative test cases.
A. STEP I: Collect Runtime Information
At ﬁrst, we collect runtime information for each test case.
Each test case for the simulator system is a pair of a user
request and a simulation result. The user request speciﬁes thetype and combination of the circuit parts in the simulation.
The simulation result includes thermal efﬁciency and their
execution time, and scripts that send and receive these kindsof information to and from the simulator. Developers can use
the scripts to reproduce the simulation result.
Our method translates the runtime behavior of a test caseinto a vector of integers with test ID. Test ID is speciﬁed
sequentially based on the timestamps original users’ test cases
are executed. Each element of a vector is the number ofoccurrences of a runtime event within the execution of thetest case. To collect runtime events from test cases, we use a
modiﬁed version of our tool SELogger version 0.2.3, which isan execution trace recorder publicly available on GitHub [8].
The tool can record the execution order of Java bytecode
instructions such as method call and local variable access. We
use the tool with the freq mode, which assigns IDs to each of
the instructions and counts their occurrences in an execution.
We simply use a vector produced by the tool as a feature
vector of the test case.
To extract only relevant runtime events to the simulator, we
exclude third party libraries from logging. This is because thetarget simulator is a web application running on Tomcat. While
the original version of SELogger records all runtime events
including the behavior of Tomcat, our version separately
collects only runtime events of the simulator. An execution
trace is separately collected for each test case.

Figure 1 shows an example of collecting runtime information. The executed instructions, consisting of runtime eventsand their location, and their execution counts are recorded as
an execution trace for each test script. In this example, theloop from line 3 to line 4 is executed three times, so someevents are repeatedly executed, such as Local Variable Loadevent, which means the value is read from the variable, at line
4. From execution traces, we create the instruction vectors
whose elements are the instruction execution counts in the
combination of the event name and location. The instruction
execution count is used in addition to the coverage so that theexecution characteristics such as a different number of loops
with the same coverage can be reﬂected in the vector. The
length of the vector in our method is the number of bytecode
instructions in the program.
B. STEP II: Classify Similar Execution
We classify vectors obtained from Step Ⅰ into groups
using the k-means clustering method. We adopt the k-means
method simply because users can specify the number oftest cases to be selected according to their time budget. In
addition, the k-means method is less computational complexity
than hierarchical clustering. We use Euclidean distance whenmeasuring between vectors. The distance distinguishes a shortsimulation from a long simulation even if they use the same
set of instructions.
After clustering, we select the oldest test case from eachcluster as the representative test cases because they are theﬁrst test cases that triggered new behaviors of the simulator.
III. C ASE S TUDY
To understand the usefulness of our method for updating
dependencies, we measure the code coverage and diversityof the execution time of test cases selected by our method.
We compare our method with three baselines: a variant of our
method using 0-1 vectors ignoring the number of occurrences,a random selection method, and a method previously used in
the company.
A. Settings
Ideally, we should run all the existing test cases andapply our method. However, the simulator system has alreadyrecorded tens of millions of test cases. It is difﬁcult to executeall of them to record the runtime information in a short periodof time due to resource constraints. Instead, we collect tendays of test cases for the case study, with a maximum of one
thousand test cases per day, and then select representative test
cases from the collected cases. The number of collected test
cases is 9,612. We decided to use k = 100 after a discussion
with developers. The developers prefer a small number in orderto ensure test cases can be executed within a short time period.
The target simulator under test has 61 classes, 509 methods,
and 21,527 lines. Our clustering method took ten minutes
on Windows 10 running a Xeon(R) W-2123 @3.60 GHz
processor, 32 GB DRAM, and an HDD.

STEPI: Collect Runtime Information

Simulator with
SELogger

Test Scripts

STEPII: Classify Similar Execution

Execution Trace

Simulator Code1: void methodA (int var) {2: if (var > 0) {3: while (var > 0)4:
var = methodB(var);
5: } else {6: var = 0;
7: }8: System.out.println(var);
9: }10: ….

Test ID

Event Name

Location

Execution
Count

Method Entry

Line 1

1

Method ParamLine 1

1

Local Variable LoadLine 2

1

Instruction Vector

Local Variable LoadLine 3

4

Test ID

Local Variable LoadLine 4

Method Call

Vector1[1,1,1,4,3,3,3,0,1,1,…]

3[1,1,1,3,2,2,2,0,1,1,…]…

…

Test ID

Vector

Select oldest
test cases
Selected ID
1

Vector2[1,1,1,0,0,0,0,1,1,1,…]

3

1 [1,1,1,4,3,3,3,0,1,1,…]

4[1,1,1,0,0,0,0,1,1,1,…]

2

Line 4

3

…

…

…

5

Local Variable StoreLine 43

Local Variable StoreLine 60

Local Variable LoadLine 81

5 [1,1,1,50,49,49,49,0,1,1,…]

Method Normal ExitLine 9

1

…

…

…

…

…

Test ID

Vector…

Fig. 1. An Overview of Our Method.

TABLE IR ESULT OF THE C OVERAGE .

Test Case Selection Method
Our clustering-based selection
Boolean-based selection (Baseline1)
Random selection (Baseline2)
Component-based selection (Baseline3)
All Test Cases Coverage

Relative Instruction Coverage
99.76%
99.88%
65.72%
97.73%
100.00%We use three baselines: Boolean-based selection, Random
selection and Component-based selection. The Boolean-based
selection is a variant of our method using a 0-1 vector, which
ignores the number of occurrences of runtime events. It is
introduced to evaluate the importance of execution counts. In
Random selection, we arranged the test cases in order when
they were executed, and selected to become chronologicallyrandom. Component-based selection is the method used bysoftware developers in Daikin Industries, Ltd. We selected test
cases so that each simulation component’s number of test cases
was as equal as developers typically do. In this case study,
we select 7 or 8 test cases for each of fourteen simulation
components, to become chronologically random and include
the oldest one same as our method, 100 in total. For eachmethod, we obtain the code coverage of selected test cases
and the distribution of the execution time.
B. Coverage Results
Table I shows the relative coverage of each method. In
this research question, we deﬁne relative coverage of a selection method as the percentage of the instructions executedby 100 test cases selected by the method divided by the
instructions executed by all 9,612 test cases. First, we can seethat the accuracy of the Component-based method is muchhigher than the Random method. This result indicates thatthe simulation component is highly related to the coverage.
Our clustering-based method and the Boolean-based method
outperform other baselines. In other words, we can select test
cases that cover different execution paths using the runtime
information. The result is better than the Component-based
method used in the company because the execution path or

Fig. 2. The Distribution of the Execution Time without Outlier.

loop counts differ depending on the parameters, even if thesimulation component is the same. On the other hand, theBoolean-based method is slightly better than our clusteringbased method. This is because our method may select test
cases with different instruction execution counts even if they
have the same instructions.
C. Execution Time Diversity Results
Figure 2 shows the distributions of the execution time
excluding outliers for each method. In this ﬁgure, outliers foreach method are removed to focus on the distribution, since
the outliers for each method are much longer than regular
executions. The longest execution of all the test cases is 328
seconds. Our method and the Random selection resulted insimilar distributions to the distribution of the original (All)test cases. On the other hand, Boolean-based method and theComponent-based method resulted in signiﬁcantly different
distributions from all test cases. By using the informationof instruction execution count, we distinguished complex
executions that take a long time from those that do not andthus achieved a diverse distribution of execution times.

Results of the Case Study
Our method using runtime information allows selecting
test cases with high coverage. By taking the number ofoccurrences into account, the method keeps the diversityof execution time.
IV. F IELD E XPERIMENT
Based on the case study results, the software developers decided to use our method for their dependency update activity.
They updated their JDK from AdoptOpenJDK 11 to Eclipse
Temurin 17 and tested their simulator using 100 test cases
selected by our method.
We received a comment from the software developers after
the update as follows: The test execution was ﬁnished in halfa day. This execution time was short enough that it did notaffect the execution of the simulator. The test cases did not ﬁnd
any incompatibilities. The simulator is working without failure
for six months until now. We considered that these selected
test cases contained various execution times and coverage,
which are better than the test cases selected by our previous
Component-based method.
From this comment, we conclude that our test case selection
method is useful for selecting the representative test cases froma large number of test cases.
In this experiment, our method did not detect any defects.
However, this does not mean that failures will not occur in the
future. The most challenging part of this research is that no one
can ensure that selected test cases are sufﬁcient for regression
testing. High code coverage does not imply a high coverage oflibrary usage scenarios implemented in the system. This might
be a reason that developers prefer 100 test cases with 99.76%
code coverage to a larger set of test cases with 100% code
coverage. An advanced method to evaluate the effectivenessof test cases in the context of dependency updates would be
helpful for software developers.
V. R ELATED W ORK
Rothermel et al. [7] and Zhang et al. [9] proposed methods
for test case selection using runtime information and sourcecode change information to ﬁnd source code that should betested. We cannot simply apply these existing methods in oursituation because the software itself does not change when thedependencies are updated.
Many researches [1] [3] [5] proposed test case selection
methods using the method call information. While our method
is close to these approaches, we adopt bytecode instructions
and their number of occurrences so that we can represent bothcode coverage and diversity of execution time as a vector.
Adithya et al. [6] proposed a system that performs datadriven test minimization. Machalica et al. [4] also proposeda new predictive test case selection strategy. Their methods
use historical test outcomes of the target software and selectthe test cases based on the predicting test results. While thesemethods are useful, they are not applicable to our simulator
because they need to maintain a history of repeatedly executed
test results for a large number of test cases.
Gligoric et al. [2] proposed a test case selection techniquethat can integrate well with testing frameworks. When developers modify test ﬁles, this technique detects and selects
the affected test cases from the dynamic dependencies. However, we could not employ only this technique because adependency update may change many ﬁles at the same time,
especially in the case of this Java update.
VI. C ONCLUSION
We report the current test case selection practice to update
dependencies in a company that maintains an industrial simulator. We found that software developers in the company useda very small number of test cases to quickly check the impact
of dependency updates, even though they had a sufﬁcientnumber of test cases achieving 100% code coverage. The
software developers recognized our method as useful because
the selected test cases included a wider range of parametersthan their previous set of test cases. The software developers
used 100 test cases selected by the method when they updated
JDK for the simulator.
In future work, we would like to combine our method with
the Component-based selection so that we can efﬁciently selecttest cases having various execution paths from tens of millionsof test cases in a practical time. We also would like to deﬁnemetrics for the diversity of execution time so that we canadopt state-of-the-art test case selection methods. We are alsointerested in how to automatically ﬁnd the optimal number ofclusters that meets developers’ requirements.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Numbers JP18H04094 and JP20H05706.
R EFERENCES[1] S. Chen, Z. Chen, Z. Zhao, B. Xu, and Y. Feng, “Using semi-supervised
clustering to improve regression test selection techniques,” in Proceedingsof the IEEE International Conference on Software Testing, Veriﬁcation
and Validation, 2011, pp. 1–10.
[2] M. Gligoric, L. Eloussi, and D. Marinov, “Practical Regression Test
Selection with Dynamic File Dependencies,” in Proceedings of theInternational Symposium on Software Testing and Analysis, 2015, pp.
211–222.
[3] E. Juergens, B. Hummel, F. Deissenboeck, M. Feilkas, C. Schlögel,and A. Wübbeke, “Regression test selection of manual system tests
in practice,” in Proceedings of the European Conference on Software
Maintenance and Reengineering, 2011, pp. 309–312.
[4] M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive test
selection,” in Proceedings of the International Conference on Software
Engineering: Software Engineering in Practice, 2019, pp. 91–100.
[5] D. Mondal, H. Hemmati, and S. Durocher, “Exploring test suite diversiﬁcation and code coverage in multi-objective test case selection,”in Proceedings of the International Conference on Software Testing,Veriﬁcation and Validation, 2015, pp. 1–10.
[6] A. A. Philip, R. Bhagwan, R. Kumar, C. S. Maddila, and N. Nagppan,“FastLane: Test Minimization for Rapidly Deployed Large-Scale Online
Services,” in Proceedings of the IEEE/ACM International Conference on
Software Engineering, 2019, pp. 408–418.
[7] G. Rothermel and M. J. Harrold, “A Safe, Efﬁcient Regression Test
Selection Technique,” ACM Transactions on Software Engineering andMethodology, vol. 6, no. 2, pp. 173–210, 1997.
[8] K. Shimari, T. Ishio, T. Kanda, N. Ishida, and K. Inoue, “NOD4J: Nearomniscient debugging tool for java using size-limited execution trace,”
Science of Computer Programming, vol. 206, 102630 (13 pages), 2021.
[9] C. Zhang, Z. Chen, Z. Zhao, S. Yan, J. Zhang, and B. Xu, “An Improved
Regression Test Selection Technique by Clustering Execution Proﬁles,” inProceedings of the International Conference on Quality Software, 2010,
pp. 171–179.