Master Thesis

Title

A Literature Review of Test Coverage Metrics

Supervisor
Prof. Shinji Kusumoto

byMasayuki Taniguchi

February 1, 2023

Departmant of Computer Science
Graduate School of Information Science and Technology
Osaka University

Master Thesis

A Literature Review of Test Coverage Metrics
Masayuki Taniguchi

Abstract
Software testing plays an essential role in software quality assurance. It helps developers to reveal andremove bugs in software. Test coverage, such as statement and branch coverage, is widely known and usedin software testing. Developers often use test coverage to measure the suﬃciency of tests, to ﬁnd untested
statements, and to localize a faulty statement. In recent years, many researchers have proposed novel metrics
for measuring test coverage of source code. However, because such novel coverage metrics are not organized,
it is impossible to understand and compare the beneﬁts and limitations of each metric. This paper organizes
the characteristics of each coverage metric by surveying a body of 80 papers that propose coverage metrics.
The survey results showed that the proposed metrics could be divided into two main groups: (1) metrics that
improve or complement traditional coverage and (2) metrics that are eﬀective in speciﬁc domains, such asconcurrent programming. We also identiﬁed the characteristics of each metric, such as eﬀective domains,
information needed to measure coverage, and granularity of measurement. Furthermore, we provide a
catalog of coverage metrics to help developers and researchers select the best metrics for their context.

Keywords
Software testing, Test coverage, Coverage metrics, Literature review

Contents
1 Introduction

1

2

Research Objective and Methodology
2.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Survey Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Test Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Target Period . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Paper Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
2
2
2
2
2

3 Research Results
3.1 RQ1: What are the objectives of the test coverage metrics proposals? . . . . . . . . . . . .
3.1.1 Backgrounds of Proposals for Coverage Metrics . . . . . . . . . . . . . . . . . . .
3.1.2 Aims of General-purpose Coverages . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.3 Aims of Domain-speciﬁc Coverages . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 RQ2: What kind of novel test coverage metrics are proposed? . . . . . . . . . . . . . . . .
3.2.1 General-purpose Coverage Metrics . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Domain-speciﬁc Coverage Metrics . . . . . . . . . . . . . . . . . . . . . . . . . .

5
5
5
5
6
6
6
8

4 Discussion
4.1 Domains of Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Available Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Evaluation of Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19
19
19
19

5 Conclusion and Future Work20

Acknowledgements

21

References

22iList of Figures
1
2

Publications per year from 1992 to 2022 . . . . . . . . . . . . . . . . . . . . . . . . . . .
Publications per venue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ii

3
4

List of Tables
1
2
3

Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Proposed Test Coverages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Available Coverage Measurement Tools . . . . . . . . . . . . . . . . . . . . . . . . . . .

iii

5
15
19

1

Introduction

Software testing is an essential activity in software quality assurance. Though software testing is a broadconcept that includes various veriﬁcation activities such as review, walkthrough and inspection, this paper
focuses on a validation activity, especially in a programmed test. The programmed test means to conﬁrm
whether the given program behaves as expected by the program execution. This paper refers to it as simply
test or testing.
Developers usually evaluate the quality of tests with many criteria. One of the most well-known criteria
is test coverage that measures the comprehensiveness of tests against source code based on the execution
path. Test coverage can be used for measuring the suﬃciency of tests, ﬁnding non-tested statements, andlocalization of a faulty statement[1].
Well-known coverages, such as statement and branch coverage, are known to have limitations. For
instance, 100% statement coverage does not guarantee that the source code has no bugs[2, 3]. To address
their drawbacks, many researchers have proposed test coverage metrics. However, these novel metrics are
not structured and organized yet. So, it is impossible to understand and compare the beneﬁts and limitationsof each metric. This lack of organization also prevents consideration of the use of such coverage metrics.
In this paper, we conduct a literature review of 80 papers on test coverage metrics. We aim to organize
the studies that propose novel coverage metrics and to examine the characteristics of each proposed metric.
Hence, our research questions are, what are the objectives of the test coverage metrics proposals? and whatkind of novel test coverage metrics are proposed?. In our analysis, we found that the proposal of coverage
metrics was primarily due to two reasons: (1) to improve or complement traditional coverage and (2) toeﬀectively measure coverage in speciﬁc domains. Based on this ﬁnding, we analyzed the characteristics ofthe coverage metrics for each proposal reason and domain. Through this study, we provide developers andresearchers with a catalog of test coverage metrics and allow them to select suitable metrics in their context.
The rest of the paper is organized as follows. In Section 2, we present our research questions, describe
the scope of our survey, and explain our methodology for collecting relevant studies. In Section 3, we show
the backgrounds of proposals for test coverage metrics and the characteristics of each metric through theanalysis of our research questions. In Section 4, we discuss future research directions on test coverage. In
Section 5, we provide ﬁnal remarks and future works.

1

2

Research Objective and Methodology

In this section, we present the two research questions that we aim to address. Following this, we describe
the scope of our survey and show our methodology to select the relevant publications for analysis.
2.1

Research Questions
Our research questions (RQs) are as follows:• RQ1: What are the objectives of the test coverage metrics proposals? This RQ explores the
backgrounds of coverage metrics proposals. We answer this RQ by investigating the problems thateach coverage metric aims to address.
• RQ2: What kind of novel test coverage metrics are proposed? In this RQ, we catalogue theproposed coverage metrics in the last few decades. This RQ provides details of each coverage metric.

2.2 Survey Scope
2.2.1

Test Coverage

Test coverage is deﬁned as a metric in software testing that measures the amount of testing performed
by a set of tests. The term test coverage is often used to refer to code coverage for white box testing (e.g.,statement and branch coverage), but this is inaccurate. In addition to code coverage, there are various
types of test coverage for diﬀerent test types and objects, such as speciﬁcation coverage[4] and mutation
coverage[5].
We focus on tests that verify the implementation of programs. Therefore, we treat the term test coverageas the coverage of tests that check the correctness of source code. Note that we are not talking about thecode coverage itself. For example, we consider both white box testing and black box testing as methods fortesting source code, although they are diﬀerent testing approaches.
2.2.2

Target Period

In 1963, Miller and Maloney[6] ﬁrst mentioned the concept of test coverage. The authors explained that
if a portion of a program is not executed by at least one test, the developer lacks the means to determine ifthat portion of code is executing correctly. Subsequently, many studies on test coverage were carried out
from the 1970s to the 1980s [7, 8, 9, 10, 11].
We are interested in the newly proposed test coverage. Therefore, we target studies that propose test
coverage metrics in the last three decades (1992-2022).
2.3

Paper Selection

We collected the papers for our study by using a speciﬁc set of keywords in some popular digital libraries.
This paper collection was performed at the beginning of May 2022.
We used the following ten keywords: test coverage, coverage metrics, code coverage, testing strategies,
software testing strategies, oracle quality, test oracle quality, test suite quality, test suite eﬀectiveness, andinsuﬃciently tested code. Since we intended to collect all papers related to our survey as much as possible,

2

9
8

Publication Count

7
6
5
4
3
2
1
0

Year

Figure 1: Publications per year from 1992 to 2022

the set of keywords includes not only those directly related to coverage metrics but also those related totesting strategy and test quality.
Using the ten keywords, we searched in the three digital libraries (ACM Digital Library, IEEE Xplore
and Google Scholar). For each query, we collected the top 200 publications in order of relevance. As aresult, we obtained 5,818 publications. (Note that a search for insuﬃciently tested code in IEEE Xplore
returned 18 publications.) After removing duplicate publications, we ended up with 4,459 publications.
Following our paper collection, we ﬁltered the publications we obtained. We ﬁrst quickly eliminatedpapers that were obviously irrelevant to our study by manually checking the titles and abstracts of all thecollected publications. Two people performed this ﬁltering to reduce the number of false negatives. This
process took two months and resulted in 237 papers. After quick ﬁltering, we conducted a full-text analysis
of each selected paper. We reviewed whether each paper proposed test coverage metrics. At the end of
this process, we obtained 43 relevant papers. Finally, we performed the forward and backward snowballing,i.e., analyzing publications citing or cited by the 43 papers. This activity took a month and produced 37
additional papers. Therefore, we ended up with a ﬁnal set of 80 papers.
We analyzed the publication years and venues of selected papers. Figure 1 shows the number of publishedpapers that propose coverage metrics per year. Until 2004, the number of papers was one or less per year;
however, the number of published papers has increased since 2005.
Figure 2 shows the number of papers per venue. We explicitly reported venues where at least two
publications were published, while we grouped all venues with only one publication together in the Other
column. We found that many venues are related to software testing, software reliability, and software
engineering. We also noticed that 13 publications (16.25%) appeared in journals, and 67 (83.75%) appearedin conferences/workshops/symposiums.

3

45
40

Publication Count

35
30

25
20
15

10
5
0

Venue

Figure 2: Publications per venue

4

3

Research Results

To answer RQ1, we examined the background of the proposed test coverage metrics for 80 studies. In
this section, we describe the results of surveying 80 studies proposing coverage metrics. It is arranged into
subsections according to the deﬁned RQs.
3.1
3.1.1

RQ1: What are the objectives of the test coverage metrics proposals?
Backgrounds of Proposals for Coverage Metrics

To answer RQ1, we examined the backgrounds of 80 studies. The results show that a common reason
for proposing coverage metrics is to address the limitations of existing coverages. We also found thateach coverage can be divided into two groups: (1) general-purpose coverages that improve or complement
traditional coverages and (2) domain-speciﬁc coverages that are eﬀective in particular domains. Generalpurpose metrics were proposed in 20 studies, and 60 studies proposed domain-speciﬁc metrics. There are
12 domains, and the breakdown of the domains is summarized in Table 1.
3.1.2

Aims of General-purpose Coverages

This section explains the aims of the proposals for general-purpose coverage metrics.
Bug Detection. One of the aims of general-purpose coverages is to enhance the ability of tests todetect bugs. For example, Aghamohammadi et al.[12] argued that incorporating the frequency of executed
statements into statement coverage leads to detecting faults more eﬀectively. McMinn et al.[13] discussed
the test coverage criteria that are well correlated with real fault revelation.
Oracle Quality. It is perfectly possible to achieve 100% coverage and still not have any result checkedby even a single oracle. In the extreme case, a test code without any assertions at all that executes every
statement in the source code achieves 100% statement coverage. Some researchers tackled this problem. For
example, Schuler and Zeller[2, 14] proposed measuring the exhaustiveness of the veriﬁcation of statementsby assertions.
Table 1: Domains
Category

Programming Paradigm

Application type

Others

DomainConcurrent Programming
Object-Oriented Programming
Multi-Staged Programming
Quantum Programming
Web Application
Service-Oriented Architecture
Database Application
GUI application
Android Application
Machine Learning Program
Fuzz Testing
Agile Development

5# papers
16
10
1
1
16
7
3
2
1
1
1
1

Test Eﬀort. Writing additional tests can be diﬃcult and costly. According to a survey of practicing
software developers, a signiﬁcant portion of this cost is due to the diﬃculty of identifying which parts of thecode should be tested[15]. Several coverage metrics have been proposed to eﬀectively identify entities to betested. For example, Huo and Clause[16] proposed direct and indirect coverage that identify how entities arecovered by tests. The authors demonstrated that faults in code indirectly tested are signiﬁcantly less likely
to be detected than those in code directly tested.
3.1.3

Aims of Domain-speciﬁc Coverages

In this section, we discuss the issues addressed by domain-speciﬁc coverage metrics.
Covering domain-speciﬁc entities. Most domain-speciﬁc coverage focuses on determining whether
the test covers domain-speciﬁc elements. Examples of domain-speciﬁc elements include thread synchronization[17] and interleaving[18] in concurrent programs, inheritance[19] and polymorphism[20] in objectoriented programs, and SQL statements[21] and DOM elements[22] in web applications.
More detailed information on the elements that each domain-speciﬁc coverage metric targets is provided
in Section 3.2.
Diﬃculty in Using Existing Coverages. In a few domains, it is diﬃcult to take advantage of existing
coverage. For example, since multi-staged programs generate and run new program code at runtime, Kim
et al.[23] proposed to measure test coverage including generated code fragments detected by static analysis.
Ali et al.[24] focused on quantum programs. Testing quantum programs is challenging for several reasons,such as the probabilistic nature and diﬃculties in estimating superposition states. The authors deﬁned test
oracles to determine the passing and failing of test suites and proposed test coverage metrics based on the
inputs and outputs of a program.
3.2

RQ2: What kind of novel test coverage metrics are proposed?

This section describes the test coverage metrics proposed in the 80 studies we selected. We show thecharacteristics of each metric according to the classiﬁcation of the proposal backgrounds from our analysis
for RQ1. Note that some studies presented multiple levels of coverage measurement granularity. Table
2 summarizes the characteristics of each test coverage, including eﬀective domain, overview, information
required to measure coverage, and granularity of measurement.
3.2.1

General-purpose Coverage Metrics

Belli et al.[25] proposed a Test Segment Coverage as coverage that bridges the gap between branchcoverage and path coverage. Test segment coverage is the path coverage of each test segment (programfragment composed of one statement or a sequence of statements). By adjusting the size of the test segments,
the thoroughness of the test coverage can be adapted to the needs of the tester.
Chen et al.[26] proposed a test coverage about variables. By calculating the program slice for a variable,
we can measure the test coverage for the code associated with that variable. In other words, the tester canfocus the test quality evaluation on important variables.
Koster et al.[27] proposed a State Coverage for test oracle assessment. This coverage measures whether
variables deﬁned at code runtime are validated by assertions using control ﬂow graphs and program slicing.
In the subsequent study by Vanoverberghe et al.[28], a general deﬁnition of State Coverage was proposed.
The authors’ deﬁnition does not require a speciﬁc structure for testing and allows more dynamic state update

6identiﬁers (e.g., object identiﬁers) than nodes in the control ﬂow graph.
Schuler et al.[2, 14] proposed a Checked Coverage. The concept of checked coverage is similar to state
coverage. This coverage metric requires testers to verify that statements that read or write variables or that
can aﬀect the control ﬂow of the program are checked by assertions. The authors consider statements on a
dynamic backward slice from an assertion as checked statements.
Zaraket et al.[29] proposed a property based coverage. This coverage derives from the hypothesis that
it is more eﬀective to evaluate test suites based on their coverage of system properties than that of structural
program elements. The authors view a property as a logical expression in an assertion and annotation. By
using the property based coverage criterion, we can measure the test coverage for all the possible values thatvariables in properties can take.
Whalen et al.[30] proposed an Observable Modiﬁed Condition/Decision Coverage (OMC/DC). OMC/DC
is a version of MC/DC that incorporates the concept of observability. The authors state that an expression
in a program is observable in a test case if we can modify its value leaving the rest of the program intact,and observe changes in the output of the system. This coverage metric helps ensure that a fault encounteredwhen executing the decision propagates to a monitored variable.
Hassan et al.[31] proposed a Multi-Point Stride Coverage (MPSC). This coverage is equivalent to branchcoverage that incorporates the concept of dataﬂow coverage by taking into account the execution order of
each branch. By using a MPSC, we can more accurately predict the quality of a test suite than control ﬂow
based coverage such as branch coverage. We can also more easily measure it than dataﬂow based coveragesuch as def-use coverage[32].
Huo et al.[16] proposed Direct and Indirect coverage. The authors argue that it is useful in the management of testing resources to consider whether entities (e.g., functions, statements, and branches) werecovered directly or indirectly by tests. This is because indirectly covered entities are only peripherally
considered and are insuﬃciently tested[16].
McMinn et al.[13] proposed a fault coverage for software testing. Fault coverage is a concept in electronicengineering that refers to the percentage of faults detected by tests out of a pre-deﬁned list of faults. The
authors discuss the way to automatically generate fault coverage for software engineering by using a fault
database such as Defects4J[33].
Byun et al.[34] proposed a Flag-Use Object Branch Coverage (Flag-Use OBC). Object branch coverage(OBC), branch coverage at the object code level, has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques. However, OBC strongly depends
on diﬀerences in object code structure due to compilers and their optimizations. While OBC is a coverage
metric based only on jump instructions, Flag-Use OBC extends OBC to include many other instructions
involved in conditional behavior.
Someoliayi et al.[35] proposed a Program State Coverage. This coverage metric improves the abilityof line coverage to validate the eﬀectiveness of the test suite. The authors consider the number of distinct
program states in which each line is executed. Program state coverage is calculated by the ratio of program
states executed in a line of tests to the maximum number of program states.
Subsequently, Aghamohammadi et al.[12] proposed a Statement Frequency Coverage. Program state
coverage has some limitations, such as the need to set a maximum number of states because we cannot
predict the number of possible states and the possibility of statements with inﬁnite states during test execution.
Statement frequency coverage solves these problems by incorporating the frequency of executed statements
into the statement coverage.

7

Bollina and Gay[36] proposed a Bytecode-MCC, a new variant of Multiple Condition Coverage (MCC)for bytecode. The authors focused on masking, which occurs when one condition prevents another frominﬂuencing the output of a boolean expression. MCC is able to overcome masking within a single expression
but cannot do so when conditions are deﬁned in multiple expressions. Bytecode-MCC groups related
Boolean expressions from the bytecode, reformulates the grouping into a single complex expression, andcalculates all possible combinations of conditions within the constructed expression.
Miranda et al.[37, 38] proposed a Relative Coverage. This is a coverage measurement technique that
focuses on the test scope of testers. By focusing coverage measurement only on in-scope entities, we can
expect to improve the cost-eﬀectiveness of testing. The authors also proposed four instances of relative
coverage: Operational Coverage[37], Social Coverage[37, 39], Relevant Coverage[37, 40], and Reachability
Coverage[38]. Operational coverage focuses on the operations performed by a speciﬁc user group. Relevant
coverage measures test coverage in the scope of testing reused code. Reachability coverage targets the input
domain that a speciﬁc user is expected to exercise. Social coverage is a coverage metric for Service-Oriented
Architecture (SOA) and will be described in Section 3.2.2.
Cox[41] proposed a Diﬀerential Coverage. This is a concept of classifying coverage information into 12
categories (newly added code is not tested, previously unused code is covered now, etc.) by comparing thecurrent version of the code with a baseline. Especially in large-scale development, the analysis of coverage
information is very costly. We can reduce the cost of coverage analysis by automatically classifying coverage
information using diﬀerential coverage.
Kolchin and Potiyenko[42] proposed a Required k-Use Chains Coverage. A k-use chain represents asequence of k-length def-ues pairs (pairs of variable deﬁnitions and uses). Existing dataﬂow coverages do
not directly require subpaths that pass a possible sequence of variable uses in conditions from a def-point
to a use-point directly associated with the def-use pair. The proposed metrics measure the test coverage ofsequences of def-use pairs.
3.2.2

Domain-speciﬁc Coverage Metrics

Concurrent Programming. Taylor et al.[43] pioneered a hierarchy of concurrency coverage criteria.
The authors extended the notion of structural testing coverage criteria to concurrent programs. The proposedcoverage metrics require tests to cover the paths of the concurrency state graph, in which the nodes represent
states of concurrent programs.
Cheer-Sun et al.[44] proposed All-du-path Coverage for parallel programs. This metric considers dupairs, triples (var, d, u) where var is a shared variable, d is a node in the parallel program ﬂow graph (PPFG)
where the value of var is deﬁned, and u is a node in the PPFG where the value is read. In other words, thismetric requires that a set of paths in which no node redeﬁnes the value of var is covered by tests.
Bron et al.[17] proposed a Synchronization Coverage. This is a practical coverage based on the idea thatcoverage tasks should be well understood by users and be coverable by tests. This coverage is accepted byIBM. Synchronization coverage has seven synchronous processes as coverage tasks.
Lu et al.[18] proposed ﬁve concurrent program interleaving coverage metrics. A fundamental problem
of concurrent program bug detection and testing is that the interleaving space is too large to be thoroughly
explored. These interleaving coverage metrics are practical and eﬀective for systematically exploring
interleaving spaces and eﬀectively ﬁnding concurrency bugs.
Trainin et al.[45] proposed coverage metrics for the detection of concurrent bugs. The proposed metrics
focus on the use of synchronization primitives (for example, synchronization processes deﬁned using the8Java keyword synchronized) and do not directly consider thread interleavings. The authors referred to
two test coverage of synchronization block and synchronization pair.
Sherman et al.[46] proposed coverage metrics inspired by Synchronization Coverage[17]. These metrics
are designed for saturation-based testing in concurrent programs; hence there is no need to estimate theexecutable domain of each metric. The authors use a combination of three basic concurrency metrics andsix contexts as coverage tasks.
Křena et al.[47] proposed coverage metrics for saturation-based and search-based testing to reﬂect
concurrency behavior accurately. In previous work[46], the identiﬁcation of elements was too roughbecause Java types were used to identify threads. The proposed metrics more accurately distinguish the
behavior of objects and threads based on object identiﬁers and thread identiﬁers. The authors derived 11
coverage metrics from dynamic analysis designed for discovering bugs in concurrent programs.
Yu et al.[48] proposed a coverage metric based on a set of interleaving idioms. An interleaving idiom
is a pattern of inter-thread dependencies and the associated memory operations. In addition, interleaving
idioms can represent more general interleaving patterns.
Tasiran et al.[49] proposed a Location-Pairs Coverage for shared-memory concurrent programs. A
location-pair is a pair of two control locations that can access the same shared variable and can be executedby two diﬀerent threads. The test coverage of location-pairs directly corresponds to the atomicity andreﬁnement violations.
Steenbuck and Fraser[50] proposed a concurrency coverage metric. The authors focused on concurrent
executions of combinations of shared memory access points with diﬀerent schedules. Therefore, this metricrequires covering all possible schedules for sets of threads, variables, and synchronization points.
Tasharoﬁ et al.[51] proposed coverage metrics for concurrent programs based on the actor model[52]. In
actor programs, concurrent entities communicate asynchronously by exchanging messages. The proposed
metrics require covering all the sequence of receive events in the program execution at three levels.
Terragni et al.[53] proposed a Sequential Coverage. This coverage metric has a sequence of events(write/read object ﬁelds, acquire/release locks and enter/exit methods) as a coverage task. We can measure
this coverage by a single thread execution of a call sequence.
Choudhary et al.[54] proposed a coverage metric based on the set of pairs of methods that executeconcurrently. Enumerating all possible interleavings (i.e., total orders of memory accesses) is practicallyinfeasible. Concurrent method pairs yield an eﬀective approximation of possible and covered interleavings.
Therefore, this test coverage metric leads to eﬃcient and practical interleaving coverage
Wang et al.[3] proposed a MAP-coverage. This coverage is based on memory-access patterns (MAP),
which are patterns of how shared variables are accessed by multiple threads[55]. MAP has often been shown
to be associated with the nature of multi-threaded bugs[55]. Thus, comprehensive testing of all MAP iseﬀective in ﬁnding bugs.
Guo et al.[56] proposed a All Synchronization Pairs Coverage. A synchronization pair is a pair ofsynchronization statements, such as synchronize block, wait(), and notify() of an object instance.
The proposed metric measures the test coverage of synchronization pairs. Although there are too many
interleavings in a multi-thread program with a high number of threads, this coverage allows interleavings to
be checked eﬃciently.
Taheri and Gopalakrishnan et al.[57] proposed coverage metrics for concurrency in GO language. The
existing concurrency coverage metrics are primarily in the context of Java and C/Pthreads. They are
not necessarily applicable to languages like Go, as such languages have diﬀerent concurrency primitives

9and semantics. The authors proposed ﬁve coverage metrics that characterize the dynamic behavior of
concurrency primitives.
Object Oriented (OO) Programming. Hsia et al.[58] proposed coverage metrics based on Enumerate
Data Member (EDM). A class is said to satisfy EDM property if its state-related data members are ofenumerate type (e.g., type enum). The authors argued that each set of values assigned to the object should
be covered by at least one test for classes satisfying EDM property. They provided three levels of coverage
metrics.
Chen et al.[59] proposed Object-Flow Coverage criteria. The proposed metrics are based on bindingof the possible classes with the program variables and object def-use pairs (pairs of object deﬁnitions anduses). We can identify object bindings and def-use pairs by using an object control ﬂow graph whose nodes
are statements that deﬁne or use an object.
Alexander et al.[60] proposed Coupling-Based Coverage metrics at three levels. These metrics focus oncoupling relationships among procedures and measure the test coverage of coupling paths (paths between
deﬁnition and use of state variables) of objects having dynamic binding.
Fisher et al.[61] proposed Change-Based Coverage metrics. The authors focused on the impact of codechanges based on the assumption that a disproportionate number of faults are likely to be present in recently
modiﬁed codes. This study deﬁnes four test coverage metrics for changed and added entities (e.g., methods
and statements) in the context of OO.
Najumudheen et al.[20] proposed criteria for method call, inheritance, and polymorphic coverage of
OO programs. These metrics are deﬁned on the Call-based Object-Oriented System Dependence Graph(COSDG), a synthesis of the dependence graph, call graph, and ﬂow graph. COSDG incorporates details
such as class, dependence, ﬂow, call, inheritance, and polymorphism.
Baldini et al.[62] proposed coverage metrics based on the program dependencies of methods. The
authors introduced a Def/Use table that represents deﬁne-use pairs of methods based on the data membersof every class. The proposed metrics measure the test coverages of all elements in the Def/Use table at threelevels.
Biswas[19] proposed Control Dependence Inheritance Coverage metrics based on JSysDG (Java System
Dependency Graph). Each time a tested class is reused through inheritance, we must retest it under a
new usage context[63]. Therefore, the cost of testing OO software can signiﬁcantly exceed that of testingprocedural programs. By using these metrics, we can eﬀectively measure the test coverage of control
dependencies associated with inheritance.
Mukherjee et al.[64,65] proposed coverage metrics in response to the fact that structural coverage metrics
for integration testing of OO programs have been scarcely reported. These coverage metrics are based ondata and control dependencies in the classes being integrated deﬁned on JSysDG.
Mukherjee[66] also focused on testing safety-critical software, such as nuclear power plants, in the OO
paradigm. Safety-critical software requires thorough testing; however, traditional coverage metrics suﬀer
from several shortcomings. The authors proposed test coverage metrics that cover program dependencies
more robustly and can detect faults at inter-object data dependencies.
Multi-Staged Programming. Kim et al.[23] proposed a New Decision Coverage for multi-staged
language. Multi-staged language is a programming language that can generate and execute new programcodes in execution time. Because it is hard to estimate what code fragments would be generated andexecuted in multi-staged language, traditional coverage is not suitable for multi-stage languages. New
decision coverage metric measures the test coverage of both branches that already exist in the program and10those generated at runtime. In the study, this metric is designed for a two-staged language.
Quantum Programming. Ali et al.[24] proposed input/output coverage metrics for the quantum
program. Testing quantum programs is diﬃcult due to the inherent characteristics of quantum computing,such as the probabilistic nature and computations in superposition. However, automatic and systematic
testing is necessary to guarantee the correct operation of quantum programs. These proposed metrics are
based on the inputs and outputs of the quantum program and measure the comprehensiveness of tests without
destroying the superpositions of the quantum program.
Web Application. Sampath et al.[67] proposed coverage criteria intended for testing web applications
at a page level. The authors deﬁned URL coverage at nine levels. A URL consists of a base address andpossible pairs of input ﬁeld names and their values as per the RFC deﬁnition <protocol>://<host> [:<port>][<path> [? <query>]][68]. The coverages report the static pages (URLs) visited during test execution for agiven web application by using a user session.
Bai et al.[69] proposed test metrics based on Web Service Description Language (WSDL)[70] elements.
The authors considered that testing of web services should cover four elements: parameters, messages (input
and output), operations, and operation ﬂows.
Smith et al.[21] proposed a SQL statement coverage for SQL injection input validation testing. Traditional coverage metrics cannot highlight how well the system protects itself through validation. SQL
statement coverage metrics measure the test coverage of SQL statements or input variables of SQL statements. Coverage data based on these metrics can provide speciﬁc information about insuﬃcient or missing
input validation.
Jokhio et al.[71] proposed two speciﬁcation based coverage metrics: boundary coverage and transition
rules path coverage. Boundary coverage refers to boundary conditions such as minimum and maximum
values for parameters. Transition rules path coverage refers to the diﬀerent execution paths that the program
may follow when receiving a given request. These are based on the goal speciﬁcation that speciﬁes the user
objectives.
Dao et al.[72] proposed coverage metrics for security testing of web applications. In many real web
applications, security sensitive sinks (e.g., functions that write data) are wrapped by wrapper functions andcalled indirectly through these wrappers. The authors proposed three levels of coverage metrics for security
sinks.
The authors also proposed a Security Sensitive Data Flow Coverage[73]. They considered that test
cases should cover as many security sensitive data ﬂows as possible to get high vulnerability detectioneﬀectiveness. This metric measures the test coverage of security sensitive branches that contain calls tosecurity sensitive sinks or propagation of the values of security sensitive variables through assignments.
Alalﬁ et al.[74] proposed coverage metrics for dynamic web applications. Faults in web applications
are often caused by insuﬃcient test coverage of complex interactions between components. These coverage
metrics are based on the client and database interactions and require testing server pages, SQL statements,and server environment variables.
Alshahwan and Harman[75] proposed coverage metrics based on HTML output uniqueness. The authors
considered that raising the diversity of the output could lead to test suites that are more eﬀective at exposing
faults. The proposed metrics measure the test coverages of unique outputs at four levels. In a subsequent
study[76], the authors extended their previous work[75] and proposed three additional coverage metrics.
Sakamoto et al.[77] proposed a Template Variable Coverage for a web application that is generated using
template engines. Template variable coverage focuses on the variables and expressions for embedding in

11

HTML templates, which are important for testing a web application’s functionality related to the dynamic
content of an HTML document. This metric measures the test coverage of template variables.
Zou et al.[78] proposed a Hybrid Coverage criteria, which combines HTML element coverage withstatement coverage of the code. The authors focused on richer iterations between client-side and server-side
in dynamic web applications. Thus, the key idea of hybrid coverage is to combine the runtime client-side
and server-side features of the web application. This coverage metric requires that tests cover all HTMLelements and statements.
Zou et al.[79] also proposed a Virtual DOM (V-DOM) Coverage. This coverage metric is based on a
V-DOM tree produced by a server script, a logical aggregation of all the possible DOM trees that represent
pages on the client side. By measuring the test coverage of the V-DOM element in the V-DOM tree, we canconsider executions on both server-side and client-side.
Mirzaaghaei et al.[22] proposed DOM Coverage metrics. Web application tests generally interact with
the DOM. The authors argued that the DOM itself should be considered as an important structure of thesystem that needs to be adequately covered by tests. Based on this idea, this paper proposed six coverage
metrics related to the DOM state.
Ed-douibi et al.[80] proposed coverage metrics for REST API. REST APIs are commonly described using
languages such as the OpenAPI1 Speciﬁcation (OAS). The proposed metrics measure the test coverages ofAPI elements based on the OAS at four levels.
Martin-Lopez et al.[81] proposed coverage metrics for RESTful API because there are no standardized
coverage criteria for black-box testing of RESTful API. These metrics measure the test coverage of elements
related to API requests/responses. The paper provides four levels of coverage metrics for each of the API
requests and responses.
Nguyen et al.[82] proposed coverage metrics for output-oriented testing of a dynamic web application.
These coverage metrics measures test coverage of string literals output and decisions that aﬀect the output.
Using these metrics helps to identify presentation faults such as HTML validation errors and spelling errors.
Service-Oriented Architecture (SOA). Mei et al.[83] proposed coverage metrics for WS-BPEL[84]applications (a kind of service-oriented application). WS-BPEL applications use XPath extensively tointegrate loosely-coupled workﬂow steps. The authors presented test coverage of the XPath query at threelevels. These metrics aim to identify defects in service compositions that are caused by faulty (or ambiguous)
XPath expressions selecting a diﬀerent XML element at runtime than the one that the composition developer
intended to be selected.
Bartolini et al.[85, 86] developed the notion of Relative Coverage that takes into consideration how theservice is used by the client. Relative coverage adapts to a tester’s context and measures the ratio between the
covered entities and those that are considered relevant in the given situation and environment. The authors
also proposed WSDL[70] based coverage metrics at three levels[87]. An interface of SOA is generally
speciﬁed in WSDL. These metrics measure the test coverages of service interfaces.
Hummer et al.[88] proposed a k-Node Dataﬂow Coverage to signiﬁcantly reduce the search space ofservice combinations in the integration test of dynamic composite Service-Based Systems (SBSs). This
metric is based on the dataﬂow of service composition. By restricting the paths for coverage measurement
to all k-length paths in the dependency tree, where a service composition is considered as a node, we canreduce the number of dataﬂows to be covered by tests.
In 2014, Miranda et al.[37, 38, 39] proposed a Social Coverage. This is the instance of relative cov1https://www.openapis.org/

12erage[37, 38]. Social coverage was conceived for black-box environments having some notion of testing
community (i.e., several users/programs using/testing the service under test). This metric measures test
coverage for the in-scope entities identiﬁed by information about the entities invoked by similar users in thesame test community. The authors assume that the service provider will measure this coverage and provide
it to the customers.
Sneed et al.[89] proposed coverage metrics based on the structure and content of the service interface.
These are test coverage of input/output parameters or combinations of parameters in input/output messages.
Using these metrics, testers can evaluate test quality without considering the source code in SOA, where
they cannot access the source code of the service under test.
Database Application. Cabal et al.[90] proposed test coverage of SQL SELECT queries. This coverage
metric requires a coverage tree built from each SELECT statement encoding the conditions speciﬁed by thewhere and join clauses of the query. This means that 100% coverage is achieved when the entire coveragetree is covered.
Willmor and Embury[91] proposed eight test coverage metrics for database applications. These coverage
metrics focus on the structural and data-oriented elements of database systems. By using the proposed
metrics, we can check the test coverage of the structural aspects of the database application (e.g., operations,transaction statements, and entities represented in the database) and the possible def-use pairs of databasesystem operations.
Halfond and Orso[92] proposed a Command-Form Coverage that is focused on the application-databaseinteractions. Most database applications dynamically generate commands in the database language (usually
SQL), pass these commands to the database for execution and process the results returned by the database.
Thus, this coverage metric measures the test coverage of all the possible SQL command forms that can beissued at each database interaction point.
GUI Application. Memon et al.[93] proposed coverage metrics for GUI testing. The input to a GUI
consists of a sequence of events. The proposed metrics thus focus on events in the structure of the GUI andmeasure the comprehensiveness of testing for events and event sequences.
Zhao and Cai[94] proposed GUI coverage metrics based on the event handlers in the source code of GUI
applications. The authors focused on data interaction relationships between event handlers and deﬁned twocoverage metrics. These metrics measure the test coverages of event handlers and deﬁnition-use handler
pairs (pairs of a handler that sets a variable and one that uses its variable.).
Android Application Jabbarvand et al.[95] proposed an eCoverage. This study aims to reduce the number of tests in energy testing of Android applications. eCoverage takes into account the energy consumptionof segments (methods or system APIs). By using this metric, we can measure test coverage of energy-greedy
segments that highly contribute to the energy consumption of the application.
Machine Learning Program. Nakajima et al.[96] proposed a Dataset Coverage for Machine Learning(ML) programs. The control structure of the ML program is so simple that any execution of the program takes
all control paths if the input training dataset is not trivial. Dataset coverage focuses on the characteristics ofthe population distribution in the training dataset in metamorphic testing.
Fuzz Testing. Tsankov et al.[97] proposed a Semi-Valid Input Coverage for fuzz testing. Traditional
coverage metrics do not measure what fuzz testing is all about, namely, executing the system with semi-valid
inputs. Semi-valid input coverage metric measures to what extent the tests cover the domain of semi-valid
inputs, where an input is semi-valid if and only if it satisﬁes all the constraints but one.
Agile Development. Rott et al.[98] proposed a Ticket Coverage for agile development. This coverage

13unveils which of the changes made in the course of a ticket are left untested. This metric measure test
coverage of the methods that were added and changed during the implementation of a given ticket and helps
to systematically focus testing eﬀorts on changed code.

1415
General-purpose
General-purpose
General-purpose
General-purpose
General-purpose
General-purpose
General-purpose
Concurrent prog.

Bytecode-MCC[36]Relative coverage[37, 38]
Operational coverage[37]Relevant coverage[37, 40]

Reachability coverage[38]Diﬀerential coverage[41]Required k-use chains coverage[42]Concurrency coverage[43]General-purpose
General-purpose

Fault coverage [13]Flag-Use Object Branch coverage[34]General-purpose

General-purpose

Direct/Indirect coverage[16]Statement frequency coverage[12]General-purpose

Multi-point stride coverage[31]General-purpose

General-purpose

Observable MC/DC[30]Program state coverage[35]DomainGeneral-purpose
General-purpose
General-purpose
General-purpose
General-purpose
General-purpose

Coverage
Test segment coverage[25]About variables[26]State coverage[27]State coverage[28]Checked coverage[2, 14]
Property based coverage[29]Overview
Adjust thoroughness of test coverage
Focus on important variables
Assess test oracle quality
General deﬁnition of state coverage[27]Assess test oracle quality
Test coverage of properties in assertions andannotation
More rigorous than MC/DC by consideringobservability
Branch coverage considering execution order of branches
Eﬀectively identify insuﬃciently tested
methods
Coverage metrics in line with actual bugs
At object code level with low dependence oncompiler structure
More eﬀectively than line coverage with low
execution cost
Overcome several shortcomings of program
state coverage[35]Overcome the masking problem than MCC
Focus on in-scope entities
Focus on operations performed by a speciﬁc
user group
Focus on entities of reused code in new(reuse) context
Focus on input domain exercised by a speciﬁc user
Classify code coverage information into 12
categories
Cover subpaths not considered in existingdata ﬂow coverage
Extend the notion of test coverage for procedural programs to concurrent programs
Concurrency state graphSet of sequences of def-use pairs

Version history

Input domain expected to be exercised

Input domain constraints

Boolean expressions in bytecode
Usage scope
Operational proﬁleNumber of execution of each statement

Number of execution of each line

Fault database
Object code

Mapping entities to tests and methods

Execution order of branches

Observability of boolean expressions

Required information
Control ﬂow graph
Program slices
Control ﬂow graph and program slices
Control ﬂow graph and program slices
Dynamic slices
All possible values of variables in properties

Table 2: Proposed Test Coverages

State

Sequence of def-use pairs

Any*

Any*

Any*

Frequency of execution of each statement
Condition
Any*
Any*

Program state

Depends on auto-generated metric
Instruction

Any*

Sequence of branches

Condition and decision

Granularity
Path
Path
Variable validation by assertions
Variable validation by assertions
Statement validation by assertions
Value of variable in property

16
OO prog.
OO prog.
OO prog.
OO prog.
OO prog.

Coupling-based coverage[60]Change-based coverage[61]Method call coverage[20]Def-use pair coverage[62]Inheritance coverage[19]Concurrent prog.
Concurrent prog.

Concurrency coverage[50]For actor program[51]Concurrent prog.
Concurrent prog.
Concurrent prog.
OO prog.
OO prog.

Concurrent prog.

Location-pairs coverage[49]MAP-coverage[3]All synchronization pairs coverage[56]
For GO lang.[57]Based on enumerate data member[58]Object-ﬂow coverage[59]Concurrent prog.

Interleaving coverage[48]Concurrent prog.

Concurrent prog.

Concurrency behavior coverage[47]Concurrent method pairs coverage[54]Concurrent prog.

Saturation-based coverage[46]Concurrent prog.

Concurrent prog.
Concurrent prog.
Concurrent prog.

Synchronization coverage[17]Interleaving coverage[18]Synchronisation primitive coverage[45]Sequential coverage[53]DomainConcurrent prog.

CoverageAll-du-path coverage[44]Overview
Extend the notion of test coverage for procedural programs to parallell programs
Practical coverage for concurrent programs
Practical and eﬀective interleaving coverage
Focuses on the use of synchronisation primitives
No need to estimate the executable domainof each metric
Accurately identify behavior of threads thanprevious work[46]Corresponds to a more general interleaving
pattern
Directly corresponds to the atomicity andreﬁnement violations
Focus on concurrent execution schedules
Focus on schedules of receiving message
eventsMeasure coverage by a single thread execution
Leads to eﬃcient and practical interleaving
coverage
Help ﬁnd multi-threaded bugs
Check thread interleavings eﬃciently
Focus on concurrency in GO language
Reveal faults related to object states
Focus on inheritance, polymorphism and dynamic behaviour of objects
Focus on coupling relationships among procedures
Expected to be eﬀective in revealing regression faults
Consider the dependencies that are uniqueto OO programs
Focuse on deﬁne and use of data members
in classes
Consider dependencies through inheritance
JSysDG

Def/Use table

COSDG

Version history

Control ﬂow graphPossible memory-access patterns[55]Synchronization pair thread graphSet of concurrency primitives
Possible object assignments
Object control ﬂow graphSet of of concurrent method pairs

Possible method call sequences

Set of thread interleavings
All possible sequences of receive events

Set of location-pairs

Set of thread interleaving idioms

Runtime behavior of threads and objects

Runtime behavior of threads

Runtime behavior of threads
Runtime behavior of threads
Runtime behavior of threads

Required information
Parallel program ﬂow graph

Granularity

Control dependency

Def-use pair

Method call

Any*

Coupling path

Memory-access pattern[55]Synchronization pair
Concurrency primitive
Object assignment
Binding or def-use pair

Concurrent method pair

Sequence of events

Thread interleaving
Sequence of receive events

Location-pair

Interleaving idiom

Synchronization behavior

Synchronization behavior

Synchronization behavior
Interleaving
Synchronization primitive

Path

17
Web app.
Web app.
Web app.
Web app.
Web app.
Web app.

Virtual DOM coverage[79]DOM coverage[22]API element coverage[80]API request/response coverage[81]Output coverage[82]Web app.

Interaction coverage[74]Hybrid Coverage[78]Web app.

Sequrity sink coverage[72]Web app.

Web app.

Speciﬁcation-based coverage[71]Template variable coverage[77]Web app.
Web app.

WSDL-based coverage[69]SQL statement coverage[21]Web app.

Web app.

Page coverage[67]Output uniquness coverage[75, 76]

Quantum prog.

Input/Output coverage[24]Web app.

OO prog.
Two-staged lang.

For safety-critical software[66]New decision coverage[23]Security sensitive data ﬂow coverage[73]DomainOO prog.

Coverage
For integration testing[64, 65]

Overview
Detect bugs missed in scenario-based integration testing
Cover program dependencies robustly
Handle branches included in code generated
at runtime
Test coverage without destroying superpositions of quantum programs
Focus on dynamic characteristics of web applications
Focus on WSDL elements
Provide speciﬁc information about insuﬃcient or missing input validation
Based on the goal speciﬁcation that speciﬁes
the user objectives
Evaluate the quality of a test suite in revealing security vulnerabilities
Cover complex interactions between clientand database
Cover as many security sensitive data ﬂows
as possible
Eﬀectively expose faults by raising the diversity of the output
Validate the dynamic contents corresponding to the execution results
Focuse on richer iterations between clientside and server-side
Consider executions on both server-side andclient-side
Provide information about DOM in web application testing
Focus on REST API testing relying on API
speciﬁcations
Focus on elements related to API requests
and responses
Help identify presentation faults such as
HTML validation errors

Required information

Set of entities related to API requests andresponses
Possible output produced from string literals

Set of API element

DOM state ﬂow graph

V-DOM tree

Set of HTML elements and statements

Set of template variables

Possible values of outputs

Set of security sensitive branches

Instrumentation transformation

Set of security sinks

Goal speciﬁcation

Set of WSDL elements
Set of sql statements or input variables

User session

Set of valid input/output values

JSysDG
Code fragments generated in execution

JSysDG

Entities related to API requests andresponses
String literal or decision

API element

DOM state or element

V-DOM

HTML element and statement

Template variable

Output value

Security sensitive branch

Entity related to interactions

Boundary condition or transition
rule path
Security sink

WSDL element
Sql statement or input variable

Web page (URL)

Input/Output value

Def-use pair
Decision

Granularity
Data and control dependency

18
Agile develop.

Ticket coverage[98]Overview
Identifying service composition faults
caused by XPath expressions
Focus on entities in test scope
Focus on service interfaces
Reduce the search space of service combinations in integration test
Focus on operations of interest to testers of
SOA-based systems
Evaluate test quality without considering thesource code in SOA
Help to detect faults at the level of SELECT
statements
Focus on the structural and data-oriented elements
Focus on the application-database interactions
Handle GUI event sequences that are much
more abstract than code
Focuse on data interaction relationships between event handlers
Consider the energy consumption of segments (methods or system APIs)
Evaluate test quality in terms of dataset variety
Measure how well tests covers the domainof semi-valid inputs
Help focus testing eﬀorts on changed codein a ticket* We can select any granularity (statement, branch, method, etc.)prog. = programming, app. = application, ML = Machine Learning, develop. = development

Fuzz Testing

GUI app.

GUI event coverage[93]Semi-valid input coverage[97]Database app.

Command-form coverage[92]ML programs

Database app.

For database app.

Dataset coverage[96]Database app.

SQL coverage[90]Android app.

SOA

Input/Output message coverage[89]eCoverage[95]SOA

Social coverage[37, 38, 39]

GUI app.

SOA
SOA
SOA

Relative coverage[85, 86]
WSDL-based coverage[87]k-Node dataﬂow coverage[88]GUI event handler coverage[94]DomainSOA

Coverage
XPath query coverage[83]Commits related to a ticket

Input constraints

Linearly separable dataset

Call graph of segments

Set of event handlers

Event ﬂow graphCharacter-level NFAs

Control ﬂow graphComplete input and output domains of service
Coverage tree

Test community

Test scope
WSDL speciﬁcation
Trees of data dependencies between services

Required information
XPath rewriting graph

MethodInput

Variety of datasets

Energy-greedy segment

Event handler

Event or event sequence

Database command

Structural element or def-use pairSQL SELECT query

Parameter in messages

Operation

Any*
Service interface
Dataﬂow

Granularity
XPath query4

Discussion

This section takes a high-level view of the studies covered in our survey. Through this activity, we
discuss future directions for research on test coverage metrics.
4.1

Domains of Studies

As shown in Section 2.3, the number of studies that propose test coverage metrics tends to increase
after 2005. We consider that the cause of this is the recent increase in the scale and complexity of softwaredevelopment and the diversity of software (e.g., the emergence of new programming paradigms).
We identiﬁed the domains of coverage proposal studies in our analysis for RQ1 and summarized themin Table 1. The results show that the research ﬁeld is well-established in concurrent programming, objectoriented programming, web applications, and service-oriented architecture. On the other hand, there area few studies in some domains, such as quantum programming and android applications. This analysis
suggests that there are many domains in need of eﬀective coverage metrics.
4.2

Available Tools

Many tools for test coverage measurement were developed in selected papers. However, most of themwere created for the evaluation of proposed coverage metrics and are not available. As of January 2023, only
nine tools are available. These tools are listed in Table 3. This makes it diﬃcult for developers to actually
use the proposed coverage metrics or for researchers to evaluate the coverage. Therefore, it is desirable tobuild available tools for coverage measurement.
4.3

Evaluation of Coverage

The 13 studies did not evaluate their proposed coverage metrics [13,19,23,29,38,41,42,58,64,66,69,71,
91]. Although the theoretical deﬁnitions of coverage metrics were discussed, it is challenging to determine
their actual eﬀectiveness without evaluation. From a practical standpoint, it is necessary to evaluate each
metric to accurately assess their eﬀectiveness, including their ability to measure test eﬀectiveness and thecost of using them.

Table 3: Available Coverage Measurement Tools
Tool
no name[34]gendiﬀcov[41]Maple[48]
CovCon[54]MAPTest[3]GOAT[57]
DomCovery[22]no name[80]
WebTest[82]Coverage
Flag-use OBC
Diﬀerential coverage
Interleaving coverage
Concurrent method pairs coverage
MAP-coverage
For GO language
DOM coverage
API element coverage
Output coverage

URL
https://github.com/tj-byun/object-coverage-criteria
https://github.com/henry2cox/lcov/tree/diffcov_initialhttps://github.com/jieyu/maple
https://github.com/michaelpradel/ConTeGe/tree/CovCon
https://github.com/sail-repos/Map-Coverage
https://github.com/staheri/goat
https://github.com/saltlab/DomCovery
https://github.com/opendata-for-all/api-tester
https://github.com/git1997/VarAnalysis

19

5Conclusion and Future WorkIn this paper, we conducted a survey of papers proposing test coverage metrics in the last three decades.
We analyzed 80 papers and answered two research questions to organize proposed coverage metrics. Our
ﬁrst research question considered the backgrounds of proposals for each metric. We investigated the
problems that each coverage aims to address. As a result, we found that proposed coverage metrics can
be classiﬁed into two categories: (1) general-purpose metrics that improve or complement traditional
coverage and (2) domain-speciﬁc metrics that are eﬀective in particular domains. Both metrics aim toaddress the limitations of existing coverage. Generic coverage aims to improve or complement traditional
coverage in a domain-independent context, while domain-speciﬁc coverage focuses on testing domainspeciﬁc elements. Our second research question set out to identify and organize the characteristics ofproposed test coverages. To that end, we examined and summarized the overview, the domain, the necessary
information for measurement, and the granularity of measurement for each coverage metric. A catalog ofnovel coverage metrics would help developers and researchers to select suitable metrics in their context.
Our future work includes the following:
Development of coverage measurement tools. We found that only nine of the 80 studies disclosed
the tools to the public. We consider that it is important to develop and publish the coverage measurement
tool. Creating tools and explaining their design and implementation will help developers and researchers.
Furthermore, making the tools available will assist in the actual development and facilitates the comparisonof coverage metrics.
Evaluation of coverage metrics. Thirteen studies did not perform the evaluation of the proposedcoverage metric. It is diﬃcult to consider the use of coverage metrics without assessing their ability
to measure test eﬀectiveness and the costs of using them. Therefore, we plan to evaluate the proposedcoverages from a practical standpoint. We also intend to identify the advantages and disadvantages of each
metric by conducting a comparative evaluation. This analysis will result in the determination of the optimal
coverage from multiple perspectives.

20

Acknowledgements
I am deeply grateful to Shinsuke Matsumoto, Assistant Professor, for his careful support and guidance
throughout the entire process of this study. He provided enthusiastic and attentive guidance from theconsultation stage to the writing of my thesis. In addition, I was also provided with valuable advice that will
be beneﬁcial to my future career outside of my research.
I would like to extend my sincere thanks to Professor Shinji Kusumoto. During my undergraduate years,
he served as the head of my major. As a result, I was taken care of by him throughout my entire university
life.
I would also like to express my gratitude to Professor Yoshiki Higo for his invaluable insights andguidance. His expertise and experience have been instrumental in shaping the study.
I am also thankful to all the members at Kusumoto Laboratory for their support and encouragement. I had
many enlightening conversations and discussions with them that deepened my understanding of information
technology and many other areas. I learned so much from them and am grateful for the opportunities togrow together.
I am grateful to the professors at the Graduate School of Information Science and Technology, Osaka
University, for their help in lectures and exercises leading up to this research.
Finally, I would like to express my deepest gratitude to my family for their support and encouragement
over the past 25 years.

21References[1] Hong Zhu, Patrick A. V. Hall, and John H. R. May. Software Unit Test Coverage and Adequacy. ACM
Computing Surveys, 29(4):366–427, 1997.
[2] David Schuler and Andreas Zeller. Checked coverage: an indicator for oracle quality. Software:
Testing, Veriﬁcation and Reliability, 23(7):531–551, 2013.
[3] Zan Wang, Yingquan Zhao, Shuang Liu, Jun Sun, Xiang Chen, and Huarui Lin. MAP-Coverage:
A Novel Coverage Criterion for Testing Thread-Safe Classes. In Proceedings of the IEEE/ACM
International Conference on Automated Software Engineering, pages 722–734, 2019.
[4] Michael Harder, Benjamin Morse, and Michael D Ernst. Speciﬁcation coverage as a measure of test
suite quality. MIT Lab for Computer Science, 2001.
[5] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Mutation
Testing Advances: An Analysis and Survey. 2018.
[6] Joan C. Miller and Cliﬀord J. Maloney. Systematic Mistake Analysis of Digital Computer Programs.
Communications of the ACM, 6(2):58–63, 1963.
[7] J. R. Brown and R. H. Hoﬀman. Evaluating the Eﬀectiveness of Software Veriﬁcation: Pratical
Experience with an Automated Tool. In Proceedings of the Fall Joint Computer Conference, Part I,
pages 181–190, 1972.
[8] John B. Goodenough and Susan L. Gerhart. Toward a Theory of Test Data Selection. SIGPLAN
Notices, 10(6):493–510, 1975.
[9] Martin R. Woodward Michael A. Hennell and David Hedley. On program analysis. Information
Processing Letters, 5(5):136–140, 1976.
[10] Martin R. Woodward, David Hedley, and Michael A. Hennell Hennell. Experience with Path Analysis
and Testing of Programs. IEEE Transactions on Software Engineering, SE-6(3):278–286, 1980.
[11] Phyllis Frankl and Elaine J. Weyuker. An applicable family of data ﬂow testing criteria. IEEE
Transactions on Software Engineering, 14(10):1483–1498, 1988.
[12] Alireza Aghamohammadi, Seyed-Hassan Mirian-Hosseinabadi, and Sajad Jalali. Statement frequency
coverage: A code coverage criterion for assessing test suite eﬀectiveness. Information and Software
Technology, 129:106426, 2021.
[13] Phil McMinn, Mark Harman, Gordon Fraser, and Gregory M. Kapfhammer. Automated Search
for Good Coverage Criteria: Moving from Code Coverage to Fault Coverage through Search-Based
Software Engineering. In Proceedings of the International Workshop on Search-Based Software
Testing, pages 43–44, 2016.
[14] David Schuler and Andreas Zeller. Assessing Oracle Quality with Checked Coverage. In Proceedingsof the IEEE International Conference on Software Testing, Veriﬁcation and Validation, pages 90–99,
2011.

22

[15] Ermira Daka and Gordon Fraser. A Survey on Unit Testing Practices and Problems. In Proceedingsof the IEEE International Symposium on Software Reliability Engineering, pages 201–211, 2014.
[16] Chen Huo and James Clause. Interpreting Coverage Information Using Direct and Indirect Coverage.
In Proceedings of the IEEE International Conference on Software Testing, Veriﬁcation and Validation,
pages 234–243, 2016.
[17] Arkady Bron, Eitan Farchi, Yonit Magid, Yarden Nir, and Shmuel Ur. Applications of Synchronization
Coverage. In Proceedings of the ACM SIGPLAN Symposium Principles and Practice of Parallel
Programming, pages 206–212, 2005.
[18] Shan Lu, Weihang Jiang, and Yuanyuan Zhou. A Study of Interleaving Coverage Criteria. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT
Symposium on The Foundations of Software Engineering, pages 533–536, 2007.
[19] Sourav Biswas. Proposal for Control Dependency White-Box Test Coverage Metrics for Inheritance. In
Proceedings of the International Conference on Data Science and Business Analytics, pages 155–160,
2018.
[20] E. S. F. Najumudheen, Rajib Mall, and Debasis Samanta. Test Coverage Analysis Based on an ObjectOriented Program Model. Journal of Software Maintenance and Evolution: Research and Practice,23(7):465–493, 2011.
[21] Ben Smith, Yonghee Shin, and Laurie Williams. Proposing SQL Statement Coverage Metrics. In
Proceedings of the International Workshop on Software Engineering for Secure Systems, pages 49–56,
2008.
[22] Mehdi Mirzaaghaei and Ali Mesbah. DOM-Based Test Adequacy Criteria for Web Applications. In
Proceedings of the International Symposium Software Testing and Analysis, pages 71–81, 2014.
[23] Taeksu Kim, Chunwoo Lee, Kiljoo Lee, Soohyun Baik, Chisu Wu, and Kwangkeun Yi. Test Coverage
Metric for Two-Staged Language with Abstract Interpretation. In Proceedings of the Asia-Paciﬁc
Software Engineering Conference, pages 301–308, 2009.
[24] Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. Assessing the Eﬀectiveness of Input and Output
Coverage Criteria for Testing Quantum Programs. In Proceedings of the IEEE Conference on Software
Testing, Veriﬁcation and Validation, pages 13–23, 2021.
[25] Fevzi Belli and Javier Dreyer. Program Segmentation for Controlling Test Coverage. In Proceedingsof the International Symposium on Software Reliability Engineering, pages 72–83, 1997.
[26] Zhenqiang Chen, Baowen Xu, Hongji Yang, and Huowang Chen. Test Coverage Analysis Basedon Program Slicing. In Proceedings of the IEEE Workshop on Mobile Computing Systems andApplications, pages 559–565, 2003.
[27] Kenneth Koster and David C. Kao. State coverage: a structural test adequacy criterion for behavior
checking. In Proceedings of the Joint Meeting of the European Software Engineering Conference andthe ACM SIGSOFT International Symposium Foundations of Software Engineering, pages 541–544,
2007.
23

[28] Dries Vanoverberghe, Jonathan de Halleux, Nikolai Tillmann, and Frank Piessens. State Coverage:
Software Validation Metrics beyond Code Coverage. In SOFSEM 2012: Theory and Practice ofComputer Science, pages 542–553, 2012.
[29] Fadi Zaraket and Wes Masri. Property Based Coverage Criterion. In Proceedings of the International
Workshop on Defects in Large Software Systems: Held in Conjunction with the ACM SIGSOFTInternational Symposium Software Testing and Analysis, pages 27–28, 2009.
[30] Michael Whalen, Gregory Gay, Dongjiang You, Mats P. E. Heimdahl, and Matt Staats. Observable
Modiﬁed Condition/Decision Coverage. In Proceedings of the International Conference on Software
Engineering, pages 102–111, 2013.
[31] Mohammad Mahdi Hassan and James H. Andrews. Comparing Multi-Point Stride Coverage andDataﬂow Coverage. In Proceedings of the International Conference on Software Engineering, pages
172–181, 2013.
[32] S. Rapps and E.J. Weyuker. Selecting software test data using data ﬂow information. IEEE Transactions
on Software Engineering, SE-11(4):367–375, 1985.
[33] René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A Database of Existing Faults to Enable
Controlled Testing Studies for Java Programs. In Proceedings of the International Symposium Software
Testing and Analysis, pages 437–440, 2014.
[34] Taejoon Byun, Vaibhav Sharma, Sanjai Rayadurgam, Stephen McCamant, and Mats P. E. Heimdahl.
Toward Rigorous Object-Code Coverage Criteria. In Proceedings of the International Symposium
Software Reliability Engineering, pages 328–338, 2017.
[35] Khashayar Etemadi Someoliayi, Sajad Jalali, Mostafa Mahdieh, and Seyed-Hassan MirianHosseinabadi. Program State Coverage: A Test Coverage Metric Based on Executed Program States. In
Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering,
pages 584–588, 2019.
[36] Srujana Bollina and Gregory Gay. Bytecode-Based Multiple Condition Coverage: An Initial Investigation. In Search-Based Software Engineering, pages 220–236, 2020.
[37] Breno Miranda and Antonia Bertolino. Testing Relative to Usage Scope: Revisiting Software Coverage
Criteria. ACM Transactions on Software Engineering Methodology, 29(3):1–24, 2020.
[38] Breno Miranda. A Proposal for Revisiting Coverage Testing Metrics. In Proceedings of the IEEE/ACM
International Conference on Automated Software Engineering, pages 899–902, 2014.
[39] Breno Miranda and Antonia Bertolino. Social Coverage for Customized Test Adequacy and Selection
Criteria. In Proceedings of the International Workshop on Automation of Software Test, pages 22–28,
2014.
[40] Breno Miranda and Antonia Bertolino. Improving Test Coverage Measurement for Reused Software.
In Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications,pages 27–34, 2015.

24

[41] Henry Cox. Diﬀerential coverage: automating coverage analysis. In Proceedings of the IEEE Conference on Software Testing, Veriﬁcation and Validation, pages 424–429, 2021.
[42] Alexander Kolchin and Stepan Potiyenko. Extending Data Flow Coverage to Test Constraint Reﬁnements. In Integrated Formal Methods, pages 313–321, 2022.
[43] Richard N. Taylor, David L. Levine, and Cheryl D. Kelly. Structural testing of concurrent programs.
IEEE Transactions on Software Engineering, 18(3):206–215, 1992.
[44] Cheer-Sun D. Yang, Amie L. Souter, and Lori L. Pollock. All-Du-Path Coverage for Parallel Programs.
In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, pages
153–162, 1998.
[45] Ehud Trainin, Yarden Nir-Buchbinder, Rachel Tzoref-Brill, Aviad Zlotnick, Shmuel Ur, and Eitan
Farchi. Forcing Small Models of Conditions on Program Interleaving for Detection of Concurrent
Bugs. In Proceedings of the Workshop on Parallel and Distributed Systems: Testing, Analysis, andDebugging, pages 1–6, 2009.
[46] Elena Sherman, Matthew B. Dwyer, and Sebastian Elbaum. Saturation-Based Testing of Concurrent
Programs. In Proceedings of the Joint Meeting of the European Software Engineering Conference andthe ACM SIGSOFT Symposium Foundations of Software Engineering, pages 53–62, 2009.
[47] Bohuslav Křena, Zdeněk Letko, and Tomáš Vojnar. Coverage Metrics for Saturation-Based andSearch-Based Testing of Concurrent Software. In Runtime Veriﬁcation, pages 177–192, 2012.
[48] Jie Yu, Satish Narayanasamy, Cristiano Pereira, and Gilles Pokam. Maple: A Coverage-Driven Testing
Tool for Multithreaded Programs. ACM SIGPLAN Notices, 47(10):485–502, 2012.
[49] Serdar Tasiran, M. Erkan Keremoğlu, and Kivanç Muşlu. Location Pairs: A Test Coverage Metric for
Shared-Memory Concurrent Programs. Empirical Software Engineering, 17(3):129–165, 2012.
[50] Sebastian Steenbuck and Gordon Fraser. Generating Unit Tests for Concurrent Classes. In Proceedingsof the International Conference on Software Testing, Veriﬁcation and Validation, pages 144–153, 2013.
[51] Samira Tasharoﬁ, Michael Pradel, Yu Lin, and Ralph Johnson. Bita: Coverage-guided, automatic
testing of actor programs. In Proceedings of the IEEE/ACM International Conference on AutomatedSoftware Engineering, pages 114–124, 2013.
[52] Gul Agha. ACTORS: A Model of Concurrent Computation in Distributed Systems. USA: MIT Press,
2004.
[53] Valerio Terragni and Shing-Chi Cheung. Coverage-Driven Test Code Generation for Concurrent
Classes. In Proceedings of the International Conference on Software Engineering, pages 1121–1132,
2016.
[54] Ankit Choudhary, Shan Lu, and Michael Pradel. Eﬃcient Detection of Thread Safety Violations via
Coverage-Guided Generation of Concurrent Tests. In Proceedings of the IEEE/ACM International
Conference on Software Engineering, pages 266–277, 2017.

25

[55] Sangmin Park, Richard Vuduc, and Mary Jean Harrold. UNICORN: A Uniﬁed Approach for Localizing
Non-Deadlock Concurrency Bugs. Software Testing, Veriﬁcation and Reliability, 25(3):167–190,
2015.
[56] JunXia Guo, Zheng Li, CunFeng Shi, and RuiLian Zhao. Thread Scheduling Sequence Generation
Based on All Synchronization Pair Coverage Criteria. International Journal of Software Engineeringand Knowledge Engineering, 30(01):97–118, 2020.
[57] Saeed Taheri and Ganesh Gopalakrishnan. GoAT: Automated Concurrency Analysis and Debugging
Tool for Go. In Proceedings of the IEEE International Symposium on Workload Characterization,
pages 138–150, 2021.
[58] Pei Hsia, Xiaolin Li, and David C. Kung. Class Testing and Code-Based Criteria. In Proceedings of
the Conference of the Centre for Advanced Studies on Collaborative Research, page 14, 1996.
[59] Mei-Hwa Chen and H.M. Kao. Testing Object-Oriented Programs - An Integrated Approach. In
Proceedings of the International Symposium on Software Reliability Engineering, pages 73–82, 1999.
[60] Roger Alexander, Jeﬀ Oﬀutt, and Andreas Steﬁk. Testing Coupling Relationships in Object-Oriented
Programs. Software Testing, Veriﬁcation and Reliability, 20:291–327, 2010.
[61] Marc Fisher, Jan Wloka, Frank Tip, Barbara G. Ryder, and Alexander Luchansky. An Evaluationof Change-Based Coverage Criteria. In Proceedings of the ACM SIGPLAN-SIGSOFT Workshop on
Program Analysis for Software Tools, pages 21–28, 2011.
[62] Fabrizio Baldini, Giacomo Bucci, Leonardo Grassi, and Enrico Vicario. Test Coverage Analysis
for Object Oriented Programs - Structural Testing Through Aspect Oriented Instrumentation. In
Proceedings of the International Conference on Software and Data Technologies, 2016.
[63] Dewayne E. Perry and Gail E. Kaiser. Adequate Testing and Object-Oriented Programming. Journalof Object-Oriented Programming, 2(5):13–19, 1990.
[64] Debashis Mukherjee, Dibyanshu Shekhar, and Rajib Mall. Proposal for A Structural Integration Test
Coverage Metric for Object-Oriented Programs. SIGSOFT Software Engineering Notes, 43(1):1–4,
2018.
[65] Debashis Mukherjee and Rajib Mall. An integration test coverage metric for Java programs. International Journal of System Assurance Engineering and Management, 10(4):576–601, 2019.
[66] Debashis Mukherjee. A Novel Test Coverage Metric for Safety-Critical Software. In Proceedings of
the TENCON - IEEE Region 10 Conference, pages 486–491, 2019.
[67] Sreedevi Sampath, Emily Hill, Sara Sprenkle, and Lori Pollock. Coverage Criteria for Testing Web
Applications. Computer and Information Sciences, University of Delaware, pages Technical Report
2005–17, 2005.
[68] RFC 1738: Uniform resource locators (url).
[69] Xiaoying Bai, Wenli Dong, Wei-Tek Tsai, and Yinong Chen. WSDL-based automatic test casegeneration for Web services testing. In Proceedings of the IEEE International Workshop on ServiceOriented System Engineering, pages 207–212, 2005.
26

[70] WWW Consortium. WSDL version 2.0., 2007.
[71] M. Shaban Jokhio, Gillian Dobbie, and Jing Sun. Towards Speciﬁcation Based Testing for Semantic
Web Services. In Australian Software Engineering Conference, pages 54–63, 2009.
[72] Thanh Binh Dao and Etsuya Shibayama. Coverage Criteria for Automatic Security Testing of Web
Applications. In Information Systems Security, pages 111–124, 2010.
[73] Thanh Binh Dao and Etsuya Shibayama. Security Sensitive Data Flow Coverage Criterion for Automatic Security Testing of Web Applications. In Engineering Secure Software and Systems, pages
101–113, 2011.
[74] Manar H. Alalﬁ, James R. Cordy, and Thomas R. Dean. Automating Coverage Metrics for Dynamic Web Applications. In Proceedings of the European Conference on Software Maintenance andReengineering, pages 51–60, 2010.
[75] Nadia Alshahwan and Mark Harman. Augmenting test suites eﬀectiveness by increasing output
diversity. In Proceedings of the International Conference on Software Engineering, pages 1345–1348,
2012.
[76] Nadia Alshahwan and Mark Harman. Coverage and Fault Detection of the Output-Uniqueness Test
Selection Criteria. In Proceedings of the International Symposium on Software Testing and Analysis,
pages 181–192, 2014.
[77] Kazunori Sakamoto, Kaizu Tomohiro, Daigo Hamura, Hironori Washizaki, and Yoshiaki Fukazawa.
POGen: A Test Code Generator Based on Template Variable Coverage in Gray-Box Integration Testingfor Web Applications. In Fundamental Approaches to Software Engineering, pages 343–358, 2013.
[78] Yunxiao Zou, Chunrong Fang, Zhenyu Chen, Xiaofang Zhang, and Zhihong Zhao. A Hybrid Coverage
Criterion for DynamicWeb Testing (S). In Proceedings of the International Conference on Software
Engineering and Knowledge Engineering, 2013.
[79] Yunxiao Zou, Zhenyu Chen, Yunhui Zheng, Xiangyu Zhang, and Zebao Gao. Virtual DOM Coveragefor Eﬀective Testing of Dynamic Web Applications. In Proceedings of the International Symposium
on Software Testing and Analysis, pages 60–70, 2014.
[80] Hamza Ed-douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. Automatic Generation of Test Cases
for REST APIs: A Speciﬁcation-Based Approach. In Proceedings of the International Enterprise
Distributed Object Computing Conference, pages 181–190, 2018.
[81] Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. Test Coverage Criteria for RESTful
Web APIs. In Proceedings of the ACM SIGSOFT International Workshop on Automating TEST Case
Design, Selection, and Evaluation, pages 15–21, 2019.
[82] Hung Viet Nguyen, Hung Dang Phan, Christian Kästner, and Tien N. Nguyen. Exploring output-based
coverage for testing PHP web applications. Automated Software Engineering, 26(1):59–85, 2019.
[83] Lĳun Mei, W.K. Chan, and T.H. Tse. Data Flow Testing of Service-Oriented Workﬂow Applications.
In Proceedings of the International Conference on Software Engineering, pages 371–380, 2008.

27

[84] Charlton Barreto, Vaughn Bullard, Thomas Erl, John Evdemon, Diane Jordan, Khanderao Kand, Dieter
König, Simon Moser, Ralph Stout, Ron Ten-Hove, Ivana Trickovic, Danny V. D. Rĳn, and Alex Yiu.
Web services business process execution language version 2.0: Primer. In OASIS, 2007.
[85] Cesare Bartolini, Antonia Bertolino, Sebastian Elbaum, and Eda Marchetti. Whitening SOA Testing.
In Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM
SIGSOFT Symposium on The Foundations of Software Engineering, pages 161–170, 2009.
[86] Marcelo Medeiros Eler, Antonia Bertolino, and Paulo Cesar Masiero. More testable service compositions by test metadata. In Proceedings of the IEEE International Symposium on Service Oriented
System, pages 204–213, 2011.
[87] Cesare Bartolini, Antonia Bertolino, Eda Marchetti, and Andrea Polini. WS-TAXI: A WSDL-based
Testing Tool for Web Services. In Proceedings of the International Conference on Software TestingVeriﬁcation and Validation, pages 326–335, 2009.
[88] Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. Test Coverageof Data-Centric Dynamic Compositions in Service-Based Systems. In Proceedings of the IEEE
International Conference on Software Testing, Veriﬁcation and Validation, pages 40–49, 2011.
[89] Harry M. Sneed and Chris Verhoef. Measuring test coverage of SoA services. In Proceedings ofthe International Symposium the Maintenance and Evolution of Service-Oriented and Cloud-Based
Environments, pages 59–66, 2015.
[90] María José Suárez-Cabal and Javier Tuya. Using an SQL Coverage Measurement for Testing Database
Applications. In Proceedings of the ACM SIGSOFT International Symposium on Foundations ofSoftware Engineering, pages 253–262, 2004.
[91] David Willmor and Suzanne Embury. Exploring test adequacy for database systems. UK Software
Testing Research Workshop, pages 123–133, 2005.
[92] William G.J. Halfond and Alessandro Orso. Command-Form Coverage for Testing Database Applications. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering,
pages 69–80, 2006.
[93] Atif M. Memon, Mary Lou Soﬀa, and Martha E. Pollack. Coverage Criteria for GUI Testing. In
Proceedings of the European Software Engineering Conference Held Jointly with ACM SIGSOFTInternational Symposium Foundations of Software Engineering, pages 256–267, 2001.
[94] Lei Zhao and Kai-Yuan Cai. Event Handler-Based Coverage for GUI Testing. In Proceedings of theInternational Conference on Quality Software, pages 326–331, 2010.
[95] Reyhaneh Jabbarvand, Alireza Sadeghi, Hamid Bagheri, and Sam Malek. Energy-Aware Test-Suite
Minimization for Android Apps. In Proceedings of the International Symposium Software Testing andAnalysis, pages 425–436, 2016.
[96] Shin Nakajima and Hai Ngoc Bui. Dataset Coverage for Testing Machine Learning Computer Programs. In Proceedings of the Asia-Paciﬁc Software Engineering Conference, pages 297–304, 2016.

28

[97] Petar Tsankov, Mohammad Torabi Dashti, and David Basin. Semi-Valid Input Coverage for Fuzz
Testing. In Proceedings of the International Symposium Software Testing and Analysis, pages 56–66,
2013.
[98] Jakob Rott, Rainer Niedermayr, Elmar Juergens, and Dennis Pagano. Ticket Coverage: Putting Test
Coverage into Context. In Proceedings of the Workshop on Emerging Trends in Software Metrics,
pages 2–8, 2017.

29