社団法人 電子情報通信学会
THE INSTITUTE OF ELECTRONICS,
INFORMATION AND COMMUNICATION ENGINEERS

信学技報
TECHNICAL REPORT OF IEICE.

A Systematic Review of Source Code Coverage Metrics: Preliminary Results
Masayuki TANIGUCHI† , Shinsuke MATSUMOTO† , and Shinji KUSUMOTO†† Graduate School of Information Science and Technology, Osaka University
E-mail: †{m-tanigt,shinsuke,kusumoto}@ist.osaka-u.ac.jp
Abstract Software testing plays an essential role in software quality assurance. It helps developers to reveal and remove bugs
in software. Developers often use test coverage to measure the suﬃciency of tests, ﬁnd non-tested statements, and localize afaulty statement. Traditional coverages, such as statement and branch coverage, are widely known and used. On the other hand,researchers have proposed various metrics for measuring test coverage of source code. Because such novel coverage metrics
are not organized, it is impossible to understand and compare the beneﬁts and limitations of each metric. This paper organizes
the characteristics of each coverage metric by surveying a body of 43 papers that propose coverage metrics. The survey results
showed that the novel metrics could be divided into two main groups: (1) metrics that improve or complement traditional coverageand (2) metrics that are eﬀective in speciﬁc domains, such as concurrent programming. We performed a comparative analysis to
identify the characteristics of each metric, such as beneﬁts of use, eﬀective domains, and information needed to measure coverage.
Furthermore, we provide a catalog of coverage metrics to help developers and researchers select the best metrics for their context.
Key words Software testing, Test coverage, Coverage metrics, Systematic review

1. Introduction

coverage metrics exist? and what are their characteristics?. In thispaper, we conduct a systematic review for 43 papers that proposed

Software testing is an essential activity in software quality assur-

novel coverage metrics to organize the characteristics of each metric.

ance. Though software testing is a broad concept that includes vari-

In our analysis, we found that the proposal of coverage metrics wasous veriﬁcation activities such as review, walkthrough and inspection,

primarily due to two reasons: (1) to improve or complement tradi-

this paper focuses a validation activity especially in programmed test.

tional coverage and (2) to eﬀectively measure coverage in speciﬁc

The programmed test means to conﬁrm whether the given program

domains. Based on this ﬁnding, we analyzed the characteristics of the

behaves as expected by the program execution. This paper refers it asnovel coverage metrics for each proposal reason and domain.

simply test or testing.
Developers usually evaluate the quality of test with many criterion.

2. Paper Selection

One of the most well known criteria is test coverage that measures the

In this section, we describe the procedure to select the papers forcomprehensiveness of tests against source code based on execution

our survey. First of all, we collected the papers for our study by us-

path. Test coverage can be used for measuring the suﬃciency of tests,

ing a speciﬁc set of keywords in some popular digital libraries. We

ﬁnding non-tested statements, and localization of a faulty statement

performed the paper collection at the beginning of May 2022.

[1].

We used the following ten keywords: test coverage, coverage met-

Traditional coverages, such as statement and branch coverage, arerics, code coverage, testing strategies, software testing strategies,

widely known and being used. The limitations of the coverage areoracle quality, test oracle quality, test suite quality, test suite eﬀec-

also well known. For instance, 100% traditional coverage does nottiveness, insuﬃciently tested code. Since we intended to collect all

guarantee that the source code has no bugs [2] [3]. To address theirpapers related to our survey as much as possible, the set of keywords

drawbacks, researchers have proposed various test coverage metrics.

includes not only those directly related to coverage metrics but also

However, these novel metrics are not structured and organized yet. So,

those related to testing strategy and test quality.

it is impossible to understand and compare the beneﬁts and limitations

Using the ten keywords, we searched in the three digital libraries

of each metric. This lack of organization also prevents consideration(ACM Digital Library, IEEE Xplore and Google Scholar). For eachof the use of novel coverage metrics.

query, we collected the top 200 publications in order of relevance.

Our goal is to provide developers and researchers with a catalog ofAs a result, we obtained 5,818 publications. (Note that a search fornovel test coverage metrics and to allow them to select suitable metrics

insuﬃciently tested code in IEEE Xplore returned 18 publications.)in their context. Hence, our research questions are, what kind of novel

After removing duplicate publications, we ended up with 4,459 pub—1—

lications.
Following our paper collection, we ﬁltered the publications weobtained. We ﬁrst quickly eliminated papers that were obviously ir-

ical expression in an assertion and annotation. By using property
based coverage criterion, we can measure the test coverage for all thepossible values that variables in properties can take.

relevant to our study by manually checking the titles and abstracts ofWhalen et al. [10] proposed Observable MC/DC (OMC/DC).

all the collected publications. The ﬁrst two authors performed thisOMC/DC is a version of MC/DC that incorporates the concept ofﬁltering to reduce the number of false negatives. This process took

observability. The authors state that an expression in a program is

two months and resulted in 237 papers. After quick ﬁltering, we

observable in a test case if we can modify its value, leaving the rest

conducted a full-text analysis of each selected paper. We reviewedof the program intact, and observe changes in the output of the sys-whether each paper proposed test coverage metrics. At the end of thistem. This coverage metric helps ensure that a fault encountered whenprocess, we obtained 43 relevant papers.

executing the decision propagates to a monitored variable.

3. Novel Test Coverage Metrics

Hassan et al. [11] proposed MultiPoint Stride Coverage. This coverage is equivalent to branch coverage that incorporates the concept of

By analyzing the 43 studies, we found that novel coverage metrics

dataﬂow coverage by taking into account the execution order of eachcan be divided into two groups: (1) those that improve or complement

branch. By using this coverage, we can more accurately predict thetraditional coverage metrics and (2) those that are speciﬁc to particularquality of a test suite than control ﬂow based coverage such as branchdomains. This section describes the overview and beneﬁts of the 43

coverage. We can also more easily measure it than dataﬂow based

metrics according to this classiﬁcation. Note that some metrics havecoverage such as def-use coverage.

multiple levels of coverage measurement granularity. Table 1 sum-

Huo et al. [12] proposed Direct/Indirect Coverage. The authors ar-marizes the characteristics of each metric, including eﬀective domain,gue that it is useful in the management of testing resources to consider

feature, information required to measure coverage and granularity ofwhether entities (e.g., functions, statements and branches) were cov-

measurement.

ered directly or indirectly by tests. This is because indirectly covered

3. 1 Metrics that improve or complement traditional coverage

entities are only peripherally considered and are insuﬃciently tested

Belli et al. [4] proposed Test Segment Coverage as a coverage that[12].

bridges the gap between branch coverage and path coverage. Test

McMinn et al. [13] proposed fault coverage for software testing.

segment coverage is the path coverage of each test segment (program

Fault coverage is a concept in electronic engineering that refers tofragment composed of one statement or a sequence of statements).

the percentage of faults detected by tests out of a pre-deﬁned list of

By adjusting the size of the test segments, the thoroughness of the test

faults. The authors discusses the way to automatically generate fault

coverage can be adapted to the needs of the tester.

coverage for software engineering by using a fault database such as

Chen et al. [5] proposed test coverage about variables. By calculat-

Defects4J [14].

ing the program slice for a variable, we can measure the test coverage

Byun et al. [15] proposed Flag-Use Object Branch Coverage. Ob-

for the code associated with that variable. In other words, the tester

ject Branch Coverage (OBC), branch coverage at the object code level,

can focus the test quality evaluation on important variables.

has the advantage of being programming language independent and is

Koster et al. [6] proposed State Coverage for test oracle assessment.

amenable to non-intrusive coverage measurement techniques. How-

This coverage measures whether variables deﬁned at code runtime are

ever, OBC strongly depends on diﬀerences in object code structure

validated by assertions, using control ﬂow graphs and program slicing.

due to compilers and their optimizations. While OBC is a coverage

In the subsequent study by Vanoverberghe et al. [7], a general

metric based only on jump instructions, Flag-Use OBC extends OBC

deﬁnition of State Coverage was proposed. The authors’ deﬁnition

to include many other instructions involved in conditional behavior.

does not require a speciﬁc structure for testing and allows more dy-

Someoliayi et al. [16] proposed Program State Coverage. This

namic state update identiﬁers (e.g., object identiﬁers) than nodes incoverage metric improves the ability of line coverage to validate thethe control ﬂow graph.

eﬀectiveness of the test suite. The authors considers the number of

Schuler et al. [2] [8] proposed Checked Coverage. The concept of

distinct program states in which each line is executed. Program statechecked coverage is similar to state coverage. This coverage metriccoverage is calculated by the ratio of program states executed in a line

requires testers to verify that statements that read or write variables orof tests to the maximum number of program states.

that can aﬀect the control ﬂow of the program are checked by asser-

Subsequently, Aghamohammadi et al. [17] proposed Statement

tions. The authors consider statements on a dynamic backward slice

Frequency Coverage. Program state coverage has some limitations,from an assertion as checked statements.

such as the need to set a maximum number of states because we cannot

Zaraket et al. [9] proposed Property Based Coverage. This cover-predict the number of possible states, and the possibility of statements

age derives from the hypothesis that it is more eﬀective to evaluatewith inﬁnite states during test execution. Statement frequency cover-test suites based on their coverage of system properties than that ofage solves these problems by incorporating the frequency of executed

structural program elements. The authors view a property as a log-

statements into the statement coverage.
—2—

Miranda et al. [18] [19] proposed Relative Coverage. This is a cov-

said to satisfy EDM property if its state-related data members are oferage measurement technique that focuses on the test scope of testers.

enumerate type. The authors argued that each set of values assigned to

By focusing coverage measurement only on in-scope entities, we canthe object should be covered by at least one test for classes satisfying

expect to improve the cost-eﬀectiveness of testing. The authors also

EDM property. They provided three levels of coverage metrics.

proposed four instances of relative coverage: Operational Coverage

Fisher et al. [29] proposed change-based coverage metrics. The[18], Social Coverage [18] [20], Relevant Coverage [18] [21], Reach-authors focused on the impact of code changes based on the assump-ability Coverage [19]. Operational coverage focuses on the operations

tion that a disproportionate number of faults are likely to be present in

performed by a speciﬁc user group. Relevant coverage measures test

recently modiﬁed codes. This study deﬁnes four test coverage metrics

coverage in the scope of testing reused code. Reachability coveragefor changed and added entities (e.g., methods and statements) in thetargets the input domain that a speciﬁc user is expected to exercise.

context of OO.

Social coverage is a coverage metric for Service-Oriented Architecture(SOA) and will be described in Section 3. 2.

Biswas [30] proposed Control Dependence Inheritance Coverage
metrics based on JSysDG (Java System Dependency Graph). EachCox [22] proposed Diﬀerential Coverage. This is a concept of clas-

time a tested class is reused through inheritance, we must retest it

sifying coverage information into 12 categories (newly added code

under new usage context [31]. Therefore, the cost of testing OO soft-

is not tested, previously unused code is covered now, etc.) by com-ware can signiﬁcantly exceed that of testing procedural programs. By

paring the current version of the code with a baseline. Especially inusing these metrics, we can measure eﬀectively the test coverage oflarge-scale development, the analysis of coverage information is very

control dependencies associated with inheritance.

costly. We can reduce the cost of coverage analysis by automaticallyclassifying coverage information using diﬀerential coverage.

Mukherjee et al. [32] proposed coverage metrics in response to thefact that structural coverage metrics for integration testing of OO pro-

3. 2 Domain Speciﬁc Metrics

grams has been scarcely reported. These coverage metrics are based

Concurrent Program. Bron et al. [23] proposed Synchronization

on data and control dependencies in the classes being integrated de-Coverage. This is a practical coverage based on the idea that cov-

ﬁned on JSysDG.

erage tasks should be well understood by users and be coverable by

Mukherjee [33] also focused on testing safety-critical software,

tests. This coverage is accepted by IBM. Synchronization coveragesuch as nuclear power plant, in the OO paradigm. Safety-critical soft-has seven synchronous processes as coverage tasks.

ware requires thorough testing; however, traditional coverage metrics

Sherman et al. [24] proposed coverage metrics inspired by synchro-suﬀers from several shortcomings. The authors proposed test cover-nization coverage. These metric are designed for saturation-based

age metrics that cover program dependencies more robustly and cantesting in concurrent programs, hence there is no need to estimate thedetect faults at inter-object data dependencies.

executable domain of each metric. The authors use a combination of

Web Application. Alalﬁ et al. [34] proposed coverage metrics forthree basic concurrency metrics and six contexts as coverage tasks.

dynamic web applications. Faults in web applications often causedKřena et al. [25] proposed coverage metrics for saturation-based

by insuﬃcient test coverage of complex interactions between compo-

and search-based testing to reﬂect concurrency behavior accurately.

nents. These coverage metrics are based on the client and database

In previous work [24], the identiﬁcation of elements is too rough be-interactions and require testing server pages, SQL statements andcause Java types were used to identify threads. The proposed metrics

server environment variables.

more accurately distinguish the behavior of objects and threads based

Mirzaaghaei et al. [35] proposed DOM Coverages. Web applica-on object identiﬁer and thread identiﬁer. The authors derived 11 cov-

tion test generally interact with the DOM. The authors argue that theerage metrics from dynamic analyses designed for discovering bugs

DOM itself should be considered as an important structure of the sys-

in concurrent programs.

tem that needs to be adequately covered by tests. Based on this idea,

Terragni et al. [26] proposed Sequential Coverage. This cover-this paper propose six coverage metrics related to the DOM state.

age metric has a sequence of events (write/read object ﬁelds, ac-

Nguyen et al. [36] proposed coverage metrics for output-oriented

quire/release locks and enter/exit methods) as a coverage task. We

testing of dynamic web application. These coverage metrics measures

can measure this coverage by a single thread execution of a call se-

test coverage of string literals output and decisions that aﬀect the out-

quence.

put. Using these metrics help to identify presentation faults such as

Wang et al. [3] proposed MAP-coverage. This coverage is based

HTML validation errors and spelling errors.

on memory-access patterns (MAP), which are patterns of how shared

Service-Oriented Architecture (SOA). Hummer et al. [37] pro-variables are accessed by multiple threads [27]. MAP have often been

posed k-Node Dataﬂow Coverage to signiﬁcantly reduce the searchshown to be associated with the nature of multi-threaded bugs [27].

space of service combinations in integration test of dynamic compos-

Thus, comprehensive testing of all MAP is eﬀective in ﬁnding bugs.

ite Service-Based Systems (SBSs). This metric is based on dataﬂow

Object Oriented (OO) Program. Hsia et al. [28] proposed cov-of service composition.By restricting the paths for coverage measure-erage metrics based on Enumerate Data Member (EDM). A class isment to all k-length paths in the dependency tree, where a service—3—composition is considered as a node, we can reduce the number ofing (ML) programs. The control structure of ML program is so simple

dataﬂows to be covered by tests.

that any execution of the program takes all control paths if the input

In 2014, Miranda et al. [18] [19] [20] proposed Social Coverage.

training dataset is not trivial. Dataset coverage focuses on the char-

This is the instance of relative coverage [18] [19]. Social coverage wasacteristics of the population distribution in the training dataset inconceived for black-box environments having some notion of testing

metamorphic testing.

community (i.e., several users/programs using/testing the service un-

Rott et al. [45] proposed Ticket Coverage for agile development.

der test). This metric measures test coverage for the in-scope entities

This coverage unveils which of the changes made in the course ofidentiﬁed by information about the entities invoked by similar users

a ticket are left untested. This metric measure test coverage of thein the same test community. The authors assume that the service

methods that were added and changed during the implementation

provider will measure this coverage and provide it to the customers.

of a given ticket and helps to systematically focus testing eﬀorts on

Sneed et al. [38] proposed coverage metrics based on the structurechanged code.

and content of the service interface. These are test coverage of in-

Martin-Lopez et al. [46] proposed coverage metrics for RESTful

put/output parameters or combinations of parameters in input/output

API because there is no standardized coverage criteria for black-box

messages. Using these metrics, testers can evaluate test quality with-testing of RESTful API. These metrics measure test coverage of el-out considering the source code in SOA where they cannot access toements related to API requests/responses. The paper provides four

the source code of the service under test.

levels of coverage metrics for each of the API requests and responses.

Others. Memon et al. [39] proposed coverage metrics for GUI

Ali et al. [47] proposed coverage metrics for quantum program.

testing. The input to a GUI consists of a sequence of events. The

Testing quantum programs is diﬃcult due to the inherent character-

proposed metrics thus focus on events in the structure of the GUI

istics of quantum computing, such as the probabilistic nature andand measure the comprehensiveness of testing for events and event

computations in superposition. However, automatic and systematicsequences.

testing is necessary to guarantee the correct operation of quantum

Smith et al. [40] proposed SQL statement coverage for SQL in-programs. These proposed metrics are based on inputs and outputs

jection input validation testing. Traditional coverage metrics cannotof the quantum program and measure the comprehensiveness of tests

highlight how well the system protects itself through validation. SQL

without destroying superpositions of the quantum program.

statement coverage metrics measure the test coverage of SQL statements or input variables of SQL statements. Coverage data based onthese metrics can provide speciﬁc information about insuﬃcient or
missing input validation.

4. Conclusion and Future WorkIn this paper, we conducted a survey of papers proposing novel test
coverage metrics. After analyzing 43 papers, we found that coverage

Kim et al. [41] proposed New Decision Coverage for multi-staged

metrics can be classiﬁed into two categories: (1) those that aim tolanguage. Multi-staged language is a programming language which

improve or complement traditional coverage metrics and (2) those thatcan generate and execute new program codes in execution time. Be-

are speciﬁc to a particular domain. We also identiﬁed and organizedcause it is hard to estimate what code fragments would be generated

the characteristics of each metric. A catalog of novel coverage met-

and executed in multi-staged language, traditional coverage is notrics would help developers and researchers to select suitable metrics

suitable for multi-stage languages. New decision coverage metricin their context.

measures the test coverage of both branches that already exist in the

In future work, we plan to perform backward and forward snow-

program and those generated at runtime. In the study, this metric isballing to make the survey as much comprehensive as possible. We

designed for a two-staged language.

will examine whether coverage metrics are proposed in each publica-

Tsankov et al. [42] proposed Semi-Valid Input Coverage for fuzztion cited by or citing any of the papers we have analyzed.

testing. Traditional coverage metrics do not measure what fuzz test-

Acknowledgments This research was partially supported bying is all about, namely executing the system with semi-valid inputs.

JSPS KAKENHI Japan (Grant Number: JP21H04877, JP20H04166,

Semi-valid input coverage metric measures to what extent the tests

JP21K18302, and JP21K11829)

cover the domain of semi-valid inputs, where an input is semi-valid if
and only if it satisﬁes all the constraints but one.
Jabbarvand et al. [43] proposed eCoverage. This study aims to
reduce the number of tests in energy testing of Android applications.
eCoverage takes into account the energy consumption of segments(methods or system APIs). By using this metric, we can measures
test coverage of energy-greedy segments that highly contribute to theenergy consumption of the application.
Nakajima et al. [44] proposed Dataset Coverage for Machine Learn-

References[1] H. Zhu, P.A.V. Hall, and J.H.R. May, “Software Unit Test Coverageand Adequacy,” ACM Comput. Surv., vol.29, no.4, pp.366–427, 1997.
[2] D. Schuler and A. Zeller, “Checked coverage: an indicator for oracle
quality,” Softw.: Testing, Verif. and Reliab., vol.23, no.7, pp.531–551,
2013.
[3] Z. Wang, Y. Zhao, S. Liu, J. Sun, X. Chen, and H. Lin, “MAP-Coverage:
A Novel Coverage Criterion for Testing Thread-Safe Classes,” Proc.
IEEE/ACM Int’l Conf. Autom. Softw. Eng., pp.722–734, 2019.
[4] F. Belli and J. Dreyer, “Program Segmentation for Controlling Test
Coverage,” Proc. Int’l Symp. Softw. Reliab. Eng., pp.72–83, 1997.

—4—

—5—

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Non-domain speciﬁc

Concurrent programs

Concurrent programs

Concurrent programs

Concurrent programs

Concurrent programs

OO programs

OO programs

OO programs

OO programs

OO programs

Web application

Web application

Web application

SOA

SOA

SOA

GUI

SQLi input valid.

Two-staged languages

FuzzingAndroid application

Machine learning

Agile development

Checked coverage [2] [8]Property based coverage [9]Observable MC/DC [10]MultiPoint stride coverage [11]Direct/Indirect coverage [12]Fault coverage for Software Testing [13]Flag-Use Object Branch coverage [15]Program state coverage [16]Statement frequency coverage [17]Relative coverage [18] [19]Operational coverage [18]Relevant coverage [18] [21]Reachability coverage [19]Diﬀerential coverage [22]Synchronization coverage [23]For saturation-based testing [24]From dynamic analysis [25]Sequential coverage [26]MAP-coverage [3]Based on enumerate data member [28]Change-based coverage [29]Control dependence inheritance coverage [30]

For integration testing [32]For safety-critical software [33]For dynamic web application [34]

DOM coverage [35]For output string literals [36]k-Node dataﬂow coverage [37]Social coverage [18] [19] [20]For parameters in messages [38]GUI event coverage [39]SQL statement coverage [40]

New decision coverage [41]Semi-valid input coverage [42]eCoverage [43]Dataset coverage [44]Ticket coverage [45]* We can select any granularity (statement, branch, method, etc.)

Quantum programs

Non-domain speciﬁc

State coverage [7]For quantum programs [47]Non-domain speciﬁc

State coverage [6]RESTful API

Non-domain speciﬁc

About variables [5]

For RESTful API [46]Non-domain speciﬁc

DomainTest segment coverage [4]MetricTest coverage without destroying superpositions of quantum programs

Test coverage of elements related to API requests/responses

Help focus testing eﬀorts on changed code in a ticket

Evaluate test quality in terms of dataset variety

Consider the energy consumption of segments (methods or system APIs)

Measure how well tests covers the domain of semi-valid inputs

Handle branches included in code generated at runtime

Provide speciﬁc information about insuﬃcient or missing input validation

Handle GUI event sequences that are much more abstract than code

Evaluate test quality without considering the source code in SOA

Focus on operations of interest to testers of SOA-based systems

Reduce the search space of service combinations in integration test

Help identify presentation faults such as HTML validation errors

Provide information about DOM in web application testing

Cover complex interactions between client and database

Cover program dependencies robustly

Detect bugs missed in scenario-based integration testing

Consider dependencies through inheritance

Expected to be eﬀective in revealing regression faults

Reveal faults related to object states

Help ﬁnd multi-threaded bugs

Measure coverage by a single thread execution

Accurately identify behavior of threads than previous work [24]No need to estimate the executable domain of each metric

Practical coverage for concurrent programs

Classify code coverage information into 12 categories

Focus on input domain that is expected to be exercised by a speciﬁc user

Focus on entities of reused code in new (reuse) context

Focus on operations performed by a speciﬁc user group

Focus on in-scope entities

Overcome several shortcomings of program state coverage [16]More eﬀectively than line coverage with low execution cost

At object code level with low dependence on compiler structure

Create coverage metrics in line with actual bugs

Eﬀectively identify insuﬃciently tested methods

Branch coverage considering execution order of branches

More rigorous than MC/DC

Test coverage of properties in assertions and annotation

Assess test oracle quality

General deﬁnition of state coverage [6]Assess test oracle quality

Focus on important variables

Adjust thoroughness of test coverage

FeatureRequired information

Set of valid input/output values

Set of entities related to API requests/responses

Commits related to a ticket

Linearly separable dataset

Call graph of segments

Input constraints

Code fragments generated in execution

Set of sql statements or input variables

Event ﬂow graphComplete input and output domains of service

Test community

Trees of data dependencies between services

Possible output produced from string literals

DOM state ﬂow graph

Instrumentation transformation

JSysDG

JSysDG

JSysDG

Version history

Possible object assignments

Possible memory-access patterns [27]Possible method call sequences

Runtime behavior of threads and objects

Runtime behavior of threads

Runtime behavior of threads

Version history

Input domain expected to be exercised

Input domain constraints

Operational proﬁleUsage scope

Number of execution of each statement

Number of execution of each line

Object code

Fault database

Mapping entities to tests and methods

Execution order of branches

Observability of boolean expressions

All the possible values of variables in properties

Dynamic slices

Control ﬂow graph and program slices

Control ﬂow graph and program slices

Program slices

Control ﬂow graph

Table 1: Novel Source Code Coverage Metrics
Granularity

Input/Output value

Entities related to API requests/responses

MethodVariety of datasets

Energy-greedy segment

Input

Decision

Sql statement or input variable

Event or event sequence

Parameter in messages

Operation

Dataﬂow

String literal or decision

DOM state or element

Entity related to interactions

Def-use pair

Data and control dependency

Control dependency

Any*

Object assignment

Memory-access pattern [27]Sequence of events

Synchronization behavior

Synchronization behavior

Synchronization behavior

Any*

Any*

Any*

Any*

Any*

Frequency of execution of each statement

Program state

Instruction

Depends on auto-generated metric

Any*

Sequence of branches

Condition and decision

Values of variables in property

Statement validation by assertions

Variable validation by assertions

Variable validation by assertions

Path

Path

[5] Z. Chen, B. Xu, H. Yang, and H. Chen, “Test Coverage Analysis Basedon Program Slicing,” Proc. IEEE Workshop on Mobile Comput. Systems and Applications, pp.559–565, 2003.
[6] K. Koster and D.C. Kao, “State coverage: a structural test adequacy
criterion for behavior checking,” Proc. Joint Meeting of the European
Softw. Eng. Conf. and the ACM SIGSOFT Int’l Symp. Foundations ofSoftw. Eng., pp.541–544, 2007.
[7] D. Vanoverberghe, J. deHalleux, N. Tillmann, and F. Piessens, “State
Coverage: Software Validation Metrics beyond Code Coverage,” SOFSEM 2012: Theory and Practice of Computer Science, pp.542–553,
2012.
[8] D. Schuler and A. Zeller, “Assessing Oracle Quality with Checked
Coverage,” Proc. IEEE Int’l Conf. Softw. Testing, Verif. and Valid.,
pp.90–99, 2011.
[9] F. Zaraket and W. Masri, “Property Based Coverage Criterion,” Proc.
Int’l Workshop on Defects in Large Softw. Systems: Held in Conjunction with the ACM SIGSOFT Int’l Symp. Softw. Testing and Anal.,
pp.27–28, 2009.
[10] M. Whalen, G. Gay, D. You, M.P.E. Heimdahl, and M. Staats, “Observable Modiﬁed Condition/Decision Coverage,” Proc. Int’l Conf. Softw.
Eng., pp.102–111, 2013.
[11] M.M. Hassan and J.H. Andrews, “Comparing Multi-Point Stride Coverage and Dataﬂow Coverage,” Proc. Int’l Conf. Softw. Eng., pp.172–181,
2013.
[12] C. Huo and J. Clause, “Interpreting Coverage Information Using Direct and Indirect Coverage,” IEEE Int’l Conf. Softw. Testing, Verif. andValid., pp.234–243, 2016.
[13] P. McMinn, M. Harman, G. Fraser, and G.M. Kapfhammer, “Automated Search for Good Coverage Criteria: Moving from Code Coverage to Fault Coverage through Search-Based Software Engineering,”
Proc. Int’l Workshop on Search-Based Softw. Testing, pp.43–44, 2016.
[14] R. Just, D. Jalali, and M.D. Ernst, “Defects4J: A Database of Existing
Faults to Enable Controlled Testing Studies for Java Programs,” Proc.
Int’l Symp. Softw. Testing and Anal., pp.437–440, 2014.
[15] T. Byun, V. Sharma, S. Rayadurgam, S. McCamant, and M.P.E. Heimdahl, “Toward Rigorous Object-Code Coverage Criteria,” Proc. Int’l
Symp. Softw. Reliab. Eng., pp.328–338, 2017.
[16] K.E. Someoliayi, S. Jalali, M. Mahdieh, and S.-H. MirianHosseinabadi, “Program State Coverage: A Test Coverage Metric
Based on Executed Program States,” IEEE Int’l Conf. Softw. Anal.,
Evolution and Reengineering, pp.584–588, 2019.
[17] A. Aghamohammadi, S.-H. Mirian-Hosseinabadi, and S. Jalali, “Statement frequency coverage: A code coverage criterion for assessing test
suite eﬀectiveness,” Info. and Softw. Technology, vol.129, p.106426,
2021.
[18] B. Miranda and A. Bertolino, “Testing Relative to Usage Scope: Revisiting Software Coverage Criteria,” ACM Trans. Softw. Eng. Methodol.,vol.29, no.3, pp.1–24, 2020.
[19] B. Miranda, “A Proposal for Revisiting Coverage Testing Metrics,”
Proc. IEEE/ACM Int’l Conf. Autom. Softw. Eng., pp.899–902, 2014.
[20] B. Miranda and A. Bertolino, “Social Coverage for Customized Test
Adequacy and Selection Criteria,” Proc. Int’l Workshop on Autom. ofSoftw. Test, pp.22–28, 2014.
[21] B. Miranda and A. Bertolino, “Improving Test Coverage Measurement for Reused Software,” Proc. Euromicro Conf. Softw. Eng. andAdvanced Applications, pp.27–34, 2015.
[22] H. Cox, “Diﬀerential coverage: automating coverage analysis,” Proc.
IEEE Conf. Softw. Testing, Verif. and Valid., pp.424–429, 2021.
[23] A. Bron, E. Farchi, Y. Magid, Y. Nir, and S. Ur, “Applications of Synchronization Coverage,” Proc. ACM SIGPLAN Symp. Principles andPractice of Parallel Programming, pp.206–212, 2005.
[24] E. Sherman, M.B. Dwyer, and S. Elbaum, “Saturation-Based Testingof Concurrent Programs,” Proc. Joint Meeting of the European Softw.
Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Softw. Eng.,
pp.53–62, 2009.
[25] B. Křena, Z. Letko, and T. Vojnar, “Coverage Metrics for SaturationBased and Search-Based Testing of Concurrent Software,” Runtime
Verif., pp.177–192, 2012.

[26] V. Terragni and S.-C. Cheung, “Coverage-Driven Test Code Generation
for Concurrent Classes,” Proc. Int’l Conf. Softw. Eng., pp.1121–1132,
2016.
[27] S. Park, R. Vuduc, and M.J. Harrold, “UNICORN: A Uniﬁed Approachfor Localizing Non-Deadlock Concurrency Bugs,” Softw. Test. Verif.
Reliab., vol.25, no.3, pp.167–190, 2015.
[28] P. Hsia, X. Li, and D.C. Kung, “Class Testing and Code-Based Criteria,” Proc. Conf. the Centre for Advanced Studies on Collaborative
Research, p.14, 1996.
[29] M. Fisher, J. Wloka, F. Tip, B.G. Ryder, and A. Luchansky, “An Evaluation of Change-Based Coverage Criteria,” Proc. ACM SIGPLANSIGSOFT Workshop on Program Anal. for Softw. Tools, pp.21–28,
2011.
[30] S. Biswas, “Proposal for Control Dependency White-Box Test Coverage Metrics for Inheritance,” Proc. Int’l Conf. Data Science andBusiness Analytics, pp.155–160, 2018.
[31] D.E. Perry and G.E. Kaiser, “Adequate Testing and Object-Oriented
Programming,” J. Obj. Oriented Prog., vol.2, no.5, pp.13–19, 1990.
[32] D. Mukherjee, D. Shekhar, and R. Mall, “Proposal for A Structural
Integration Test Coverage Metric for Object-Oriented Programs,” SIGSOFT Softw. Eng. Notes, vol.43, no.1, pp.1–4, 2018.
[33] D. Mukherjee, “A Novel Test Coverage Metric for Safety-Critical Software,” IEEE Region 10 Conf., pp.486–491, 2019.
[34] M.H. Alalﬁ, J.R. Cordy, and T.R. Dean, “Automating Coverage Metrics for Dynamic Web Applications,” Proc. European Conf. Softw.
Maintenance and Reengineering, pp.51–60, 2010.
[35] M. Mirzaaghaei and A. Mesbah, “DOM-Based Test Adequacy Criteria
for Web Applications,” Proc. Int’l Symp. Softw. Testing and Anal.,
pp.71–81, 2014.
[36] H.V. Nguyen, H.D. Phan, C. Kästner, and T.N. Nguyen, “Exploring output-based coverage for testing PHP web applications,” Autom.
Softw. Eng., vol.26, no.1, pp.59–85, 2019.
[37] W. Hummer, O. Raz, O. Shehory, P. Leitner, and S. Dustdar, “TestCoverage of Data-Centric Dynamic Compositions in Service-Based
Systems,” IEEE Int’l Conf. Softw. Testing, Verif. and Valid., pp.40–49,
2011.
[38] H.M. Sneed and C. Verhoef, “Measuring test coverage of SoA services,” Proc. Int’l Symp. the Maintenance and Evolution of ServiceOriented and Cloud-Based Environments, pp.59–66, 2015.
[39] A.M. Memon, M.L. Soﬀa, and M.E. Pollack, “Coverage Criteria for
GUI Testing,” Proc. European Softw. Eng. Conf. Held Jointly with
ACM SIGSOFT Int’l Symp. Foundations of Softw. Eng., pp.256–267,
2001.
[40] B. Smith, Y. Shin, and L. Williams, “Proposing SQL Statement Coverage Metrics,” Proc. Int’l Workshop on Softw. Eng. for Secure Systems,
pp.49–56, 2008.
[41] T. Kim, C. Lee, K. Lee, S. Baik, C. Wu, and K. Yi, “Test Coverage
Metric for Two-Staged Language with Abstract Interpretation,” AsiaPaciﬁc Softw. Eng. Conf., pp.301–308, 2009.
[42] P. Tsankov, M.T. Dashti, and D. Basin, “Semi-Valid Input Coverage forFuzz Testing,” Proc. Int’l Symp. Softw. Testing and Anal., pp.56–66,
2013.
[43] R. Jabbarvand, A. Sadeghi, H. Bagheri, and S. Malek, “Energy-Aware
Test-Suite Minimization for Android Apps,” Proc. Int’l Symp. Softw.
Testing and Anal., pp.425–436, 2016.
[44] S. Nakajima and H.N. Bui, “Dataset Coverage for Testing Machine Learning Computer Programs,” Asia-Paciﬁc Softw. Eng. Conf.,
pp.297–304, 2016.
[45] J. Rott, R. Niedermayr, E. Juergens, and D. Pagano, “Ticket Coverage:
Putting Test Coverage into Context,” Proc. Workshop on Emerging
Trends in Softw. Metrics, pp.2–8, 2017.
[46] A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “Test Coverage Criteria for RESTful Web APIs,” Proc. ACM SIGSOFT Int’l Workshop on
Automating TEST Case Design, Selection, and Evaluation, pp.15–21,
2019.
[47] S. Ali, P. Arcaini, X. Wang, and T. Yue, “Assessing the Eﬀectivenessof Input and Output Coverage Criteria for Testing Quantum Programs,”
Proc. IEEE Conf. Softw. Testing, Verif. and Valid., pp.13–23, 2021.

—6—