DeepGauge: Multi-Granularity Testing Criteria for Deep
Learning Systems
Lei Ma1,3∗ , Felix Juefei-Xu2 , Fuyuan Zhang3 , Jiyuan Sun4 , Minhui Xue3 , Bo Li5
Chunyang Chen6 , Ting Su3 , Li Li6 , Yang Liu3 , Jianjun Zhao4 , and Yadong Wang1

1 Harbin Institute of Technology, China 2 Carnegie Mellon University, USA 3 Nanyang Technological University, Singapore
4 Kyushu University, Japan 5 University of Illinois at Urbana–Champaign, USA 6 Monash University, Australia

ABSTRACTIn Proceedings of the 2018 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE ’18), September 3–7, 2018, Montpellier,France. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3238147.
3238202

Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron
network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethoraof studies have shown that the state-of-the-art DL systems sufferfrom various vulnerabilities which can lead to severe consequences
when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy oftest data. Considering the limitation of accessible high quality test
data, good accuracy performance on test data can hardly provide
confidence to the testing adequacy and generality of DL systems.
Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL
system makes system analysis and defect detection difficult, which
could potentially hinder its real-world deployment. In this paper,
we propose DeepGauge, a set of multi-granularity testing criteria
for DL systems, which aims at rendering a multi-faceted portrayalof the testbed. The in-depth evaluation of our proposed testing
criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques
against DL. The potential usefulness of DeepGauge sheds light on
the construction of more generic and robust DL systems.

1

INTRODUCTION

Deep learning (DL) systems have gained great popularity in various
applications, e.g., speech processing [26], medical diagnostics [12],image processing [11], and robotics [58]. A deep neural network(DNN), as a type of deep learning systems, is the key driving force
behind recent success. However, DNN-based software systems,
such as autonomous driving, often exhibit erroneous behaviors
that lead to fatal consequences. For example, several accidents [21]have been reported due to autonomous vehicle’s failure to handle
unexpected/corner-case driving conditions.
One of the trending research areas is to investigate the cause ofvulnerability in DL systems by means of generating adversarial test
examples for image- and video-based DL systems. Such carefully
learned pixel-level perturbations, imperceptible to human eyes,
can cause the DL-based classification system to output completely
wrong decisions with high confidence [20]. Ever since the inceptionof adversarial attacks on the DL systems, more and more researchhas been dedicated to building up strong attackers [6, 25, 55, 60]. Asa consequence, better defense mechanisms in DL systems against
adversarial attacks are in dire need. Various techniques to nullify
adversarial attacks and to train a more robust DL system are emerging in recent studies [18, 23, 41, 43, 45, 51, 56]. Together, researchin both realms forms a virtuous circle and blazes a trail for better
understanding of how to build more generic and robust DL systems.
However, what is still lacking is a systematic way of gauging thetesting adequacy of given DL systems. Current studies focus only
on pursuing high accuracy of DL systems as a testing criterion, forwhich we show several caveats as follows. First, measuring thesoftware quality from DL output alone is superficial in the sensethat fundamental understanding of the DL internal neuron activities
and network behaviors is not touched upon. We agree that it could
be an indicator of DL system quality and generality, but it is far
from complete, and oftentimes unreliable. Second, a criterion solely
based on DL output will rely heavily on how representative the test
data are. Having achieved high-performance DL output does notnecessarily mean that the system is utmost generic, and achieving
low-performance does not indicate the opposite either. A DL model
can be immune to many known types of adversarial attacks, but may
fail from unseen attacks. This is because such a criterion based only
on DL outputs is far from being comprehensive, and it leaves highrisks for currently cocooned DL systems to be deployed in the realworld environment where newly evolved adversarial attacks are

CCS CONCEPTS• Software and its engineering → Software testing and debugging; • Theory of computation → Adversarial learning;

KEYWORDS
Deep learning, Software testing, Deep neural networks, Testing
criteria
ACM Reference Format:
Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018.
DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems.
∗ Lei Ma is the corresponding author. Email: malei@hit.edu.cn.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
ASE ’18, September 3–7, 2018, Montpellier, France© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5937-5/18/09. . . $15.00
https://doi.org/10.1145/3238147.3238202

120

ASE ’18, September 3–7, 2018, Montpellier, France

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wang

inescapable. Third, any DL system that passes systematic testingshould be able to withstand all types of adversarial attacks to someextent. Such generality upon various attacks is of vital importancefor DL systems to be deployed. But apparently this is not the case,unless we stick to a set of more comprehensive gauging criteria.
We understand that even the most comprehensive gauging criteria
would not be able to entirely eliminate risks from adversarial attacks.
Nevertheless, by enforcing a suitable set of testing criteria, we hopethat a DL system could be better tested to facilitate the constructionof a more generic and robust deep learning system.
Towards addressing the aforementioned limitations, a set of testing criteria is needed, as opposed to the sole criterion based on
DL decision output. In addition to being scalable, the proposed
criteria will have to monitor and gauge the neuron activities andintrinsic network connectivity at various granularity levels, so thata multi-faceted in-depth portrayal of the DL system and testing
quality measures become desirable.
In this work, we are probing this problem from a software engineering and software testing perspective. At a high level, erroneous
behaviors appeared in DNNs are analogous to logic bugs in traditional software. However, these two types of software are fundamentally different in their designs. Traditional software represents
its logic as control flows crafted by human knowledge, while a DNN
characterizes its behaviors by the weights of neuron edges and thenonlinear activation functions (determined by the training data).
Therefore, detecting erroneous behaviors in DNNs is different fromdetecting those in traditional software in nature, which necessitatesnovel test generation approaches.
To achieve this goal, the very first step is to precisely define
a set of suitable coverage criteria, which can guide test designand evaluate test quality. Despite a number of criteria existing for
traditional software, e.g., statement, branch, data-flow coverage,
they completely lose effect in testing DNNs. To the best of ourknowledge, the design of testing coverage criteria for DNNs is still at
the early stage [38, 47]. Without a comprehensive set of criteria, (1)designing tests to cover different learned logics and rules of DNNs
is difficult to achieve. Consequently, erroneous behaviors may be
missed; (2) evaluating test quality is biased, and the confidence ofobtained testing results may be overestimated. In this paper, we
propose DeepGauge—a set of testing criteria based on multi-leveland -granularity coverage for testing DNNs and measure the testing
quality. Our contributions are summarized as follows:adversarial test data using 4 well-known adversarial data generation algorithms (i.e., Fast Gradient Sign Method (FGSM) [20],Basic Iterative Method (BIM) [31], Jacobian-based Saliency Map
Attack (JSMA) [37] and Carlini/Wagner attack (CW) [8]). The
extensive evaluation shows that our criteria can effectively capture the difference between the original test data and adversarial
examples, where DNNs could and could not correctly recognize,
respectively, demonstrating that a higher coverage of our criteria
potentially indicates a higher chance to detect the DNN’s defects.
• The various criteria proposed behave differently on DNNs w.r.t.
network complexity and dataset under analysis. Altogether, thesecriteria can potentially help us gain insights of testing DNNs. By
providing these insights, we hope that both software engineeringand machine learning communities can benefit from applying
new criteria for gauging the testing quality of the DNNs to gain
confidence towards constructing generic and robust DL systems.
To the best of our knowledge, this is among the earliest studies
to propose multi-granularity testing criteria for DL systems, which
are mirrored by the test coverage in traditional software testing.

2

PRELIMINARIESIn this section, we first introduce traditional software and thendeep learning systems of which the architectural features appearing
to be a step above current traditional software. We will see that
DL fundamentally changes the software development paradigm.
Precisely, we try to analogize that the programming language logic
execution to traditional software is what the connectivity strength(weights) to a DNN. As we will see below, we are attempting toconnect these two counterparts as well as discussing the differences.

2.1

Coverage Criteria in Traditional Software
Testing

We regard traditional software as any program written in high-level
programming languages (e.g., C/C++, Java, Python). Specially, eachstatement in traditional program performs some certain operationthat either transforms the outputs from the previous statement to
the next one or changes the program states (e.g., assign new values
to variables). Software defects (bugs) can be introduced by developers
due to incorrect implementation, which may cause unexpected
outputs or even fail-stop errors (e.g., program crashes).
To detect defects, software testing is one of the most widelyadopted software validation techniques in software industry— Givena set of test data, it feeds these test data as inputs to program andvalidates the correctness of the program’s run-time behavior bycomparing the actual outputs with expected ones (test oracles); andmeasures test adequacy by using coverage criteria, the important,practical measures to quantify the degree to which the software istested [36]. The program with higher test coverage often suggeststhat it has a lower chance of containing defects. Many software
testing standards require a software product to be thoroughly tested
with high test coverage before shipment, which is used as an indicator and confidence of the software quality. On some safety criticalsystems, the requirement of some form of test coverage is even
100%. For example, ECSS-E-ST-40C [15] standards demand 100%
statement coverage of the software under test for two critical levels.

• Our proposed criteria facilitate the understanding of DNNs as
well as the test data quality from different levels and angles.
In general, we find defects could potentially distribute on bothmajor function regions as well as the corner-case regions of
DNNs. Given a set of inputs, our criteria could measure to what
extent it covers the main functionality and the corner cases of theneurons, where DL defects could incur. Our evaluation results
reveal that the existing test data of a given DL in general skew
more towards testing the major function region, with relatively
few cases covering the corner-case region.
• In line with existing test data of DNNs, we evaluate the usefulness of our coverage criteria as indicators to quantify defectdetection ability of test data on DNNs, through generating new121

DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
For traditional software, a number of coverage criteria havealready been defined at different levels, to analyze the software runtime behavior from different perspectives, i.e., code level (e.g., statement, branch, data-flow coverage and mutation testing [27, 46, 62])or model-level (e.g., state and transition coverage [2, 14]) to caterfor different testing methods and granularities. Some commonlyused test coverage criteria are listed as follows:n4

All Behaviors

n1
n5

n8

n6

n9

n2

Main Behaviors(k-multisection Coverage)

Corner-case
Behaviors(Boundary Coverage)n3
n7• Statement coverage measures whether each instruction has been
executed, and branch coverage focuses on whether each branchof control structure (e.g., in if or switch-case statements) has
been covered, both of which are control-flow-based criteria.
• Data-flow coverage [46] enforces the coverage of each variable
definition and its uses to detect data-flow anomalies.
• Model-based coverage criteria [3, 52] aim to cover more program
behaviors via abstracted behavior models. Other comprehensive
variants of test coverage could be referred to [2].

Layer 1

Layer 2

Layer 3

(Input Layer)(Hidden Layer)(Output Layer)(a)Erroneous
Behaviors(b)

Figure 1: (a) An example of a fully connected DNN. (b) Behaviors of DNNs and relations between defined coverage criteria(the red points denote erroneous behaviors therein).
to define a set of suitable coverage criteria for evaluating the test
adequacy as well as gauging the internal covered states of DNNs
to gain confidence on the testing results of DNNs.

However, none of these criteria can be directly applied to test
DNNs due to its unique architecture, as explained below.

2.2

ASE ’18, September 3–7, 2018, Montpellier, France

3

Deep Neural Network Architecture

COVERAGE CRITERIA FOR TESTING DL
SYSTEMSFor traditional software testing, developers design and seek a set ofrepresentative test data from the whole large input space, hopingthat the selected test data could detect the software defects underlimited computational resources.3
Testing coverage criteria is proposed to shatter and approximate
the software internal states. It partitions the input space and establishes the relation of an input subspace and an approximatedsoftware internal state. In this way, compared with the test datafrom a single input subspace, the same number of test data fromdifferent input sub-spaces would have a higher chance to cover
more diverse software states, resulting in a higher possibility todetect more diverse software defects. Over the past decades, a set ofwell-designed coverage criteria [2] (e.g., statement coverage, branchcoverage) have demonstrated their practical value and are widelyadopted in software industry to systematically guide the testingprocess to unveil the software defects at different levels, e.g., (1) Unitlevel: testing small snippets of functions. (2) Integration level: testing
multiple sub-modules or functions to check their interactions. (3)System level: testing the software system as a whole.
The current state-of-the-practice DNN testing, however, is still at
its early stage and mainly relies on the prediction accuracy (similar
to black-box system level testing that only observes inputs andits corresponding outputs), lacking systematic testing coverage
criteria for defect detection. Furthermore, traditional software and
DNNs have obvious differences, so existing coverage criteria for
traditional software could not be directly applied to DNNs.
In this section, we design a set of DNN testing coverage criteria
from multiple levels, aiming to gauge the testing adequacy of DNNs
and facilitate the detection of those erroneous behaviors from multiple portrayals. To be useful towards industry level applications,
we believe that the test criteria should be simple, scalable as wellgeneral enough to be applied to a large range of DNNs without
confining on specific DNN structure or activation functions. Conceptually, similar to traditional software, the behaviors of DNNs
can be divided into two categories, i.e., major function behaviors

In our paper, we regard a DL system as any software system that
includes one or more DNNs.1 Unlike traditional software, programmed with deterministic algorithms by developers, DNNs areprogrammed by the training data, selected features, and network
structures (e.g., number of layers). Specially, a DNN consists ofmultiple interconnected neurons organized on layers: the input
layer, the output layer, and one or multiple hidden layers. Eachneuron is a computing unit that computes its output by applying anactivation function to its input. In classic DNNs, each neuron is fullyconnected with all neurons on the next layer, and each edge has aweight, which indicates the strength of the connections between
neurons. Overall, a DNN could be considered as a function thattransforms a given input to the output, and this function is decided
by the aggregated effects from its computation units (i.e., neurons),
each of which contributes to the whole computation procedure.
Figure 1(a) shows an example of a three-layer DNN.
To accomplish a task (e.g., prediction on the autonomous vehicles’ steering angle by monitored images), DNNs are trained andprogrammed through a large set of labelled training data. However,similar to traditional software, DNNs may also contain defects (e.g.,give wrong steering angles) due to incorrect, incomplete training
data, or even the wrongly stipulated run-time programming (i.e.,
training) procedure. For example, human analyst may include erroneous and noisy data when collecting training data. In such acase, a given input data might be wrongly handled (e.g., classified, predicted), causing losses and even severe tragedies, if theflawed DNNs are deployed to safety-critical systems (e.g., the recent Tesla autonomous driving accident2 ). For the complex andhigh-dimensional real-world inputs, it is almost impossible for human to ensure all possible, even corner-case data are included. To
systematically test and uncover hidden defects of DNNs, it is crucial
1 In particular, a DL system may either be entirely composed of DNNs, or have DNNs as its corewith extra software encapsulation. In this paper, we mostly focus on DNNs since it is the core ofa DL system, and our methods could be extended to support general DL systems. Although thetraining program of the current state-of-the-art DNNs are still written as traditional software, theobtained DNN from the training program is fundamentally different in how the logic is encoded.
2 http://www.bbc.com/news/world-us-canada-43604440

3 The input space of a software could be so large that it is often impossible to enumerate and test all

the possibilities given limited computation resource.

122ASE ’18, September 3–7, 2018, Montpellier, France

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wangand corner-case behaviors, both of which may contain erroneous
behaviors (see Figure 1(b) and our evaluation results in Section 4).
We have taken these factors into consideration during the designof coverage criteria.
Let N = {n 1 , n 2 , . . .} be a set of neurons of a DNN. Let T ={x1 , x2 , . . .} be a set of test inputs. We use ϕ(x, n) to denote a function that returns the output of a neuron n ∈ N under a given test
input x ∈ T .4 Let the DNN have l layers and Li denote the set ofneurons on the i-th layer (1 ≤ i ≤ l).

3.1

We further define the k-multisection neuron coverage of a DNN as:Ínnn ∈N |{S i | ∃x ∈ T : ϕ(x, n) ∈ S i }|KMNCov(T , k) =.
k × |N |However, for a neuron n, there are also cases where ϕ(x, n) may
locate out of [lown , highn ], i.e., ϕ(x, n) ∈ (−∞, lown ) or ϕ(x, n) ∈(highn , +∞). We refer to (−∞, lown ) ∪ (highn , +∞) as the cornercase region of a neuron n.
Definition 3.2. For a test input x ∈ T , we say that a DNN islocated in its corner-case region given x iff ∃n ∈ N : ϕ(x, n) ∈(−∞, lown ) ∪ (highn , +∞).

Neuron-Level Coverage Criteria

At the neuron-level, we use the output values of neuron n determined from the training to characterize its behaviors. Since theinternal logic of a DNN is mostly programmed by training data,
intuitively, the functionality (i.e., neuron output) for each neuron of
a DNN should follow some statistical distribution that is largely determined by the training data. The output distribution of a neuron
obtained from training data analysis would allow to approximately
characterize the major function regions whose output values areoften triggered by input data with a similar statistical distribution to
the training data, and the corner cases whose output values rarely
occur. However, for a practical-sized DNN, obtaining an accurate
output distribution for each neuron would be computationally intensive. With the similar spirit while being scalable, we leverage
the neuron output value boundaries obtained from training data toapproximate the major function region and corner-case region.
Specially, for a neuron n, let highn and lown be its upper andlower boundary output values, respectively, on the value range of
its activation function, where highn and lown are derived from thetraining dataset analysis. We refer to [lown , highn ] as the major
function region of a neuron n.

Note that the profiled outputs of a neuron obtained from thetraining data would not locate into the corner-case region.In otherwords, if test inputs follow a similar statistical distribution withthe training data, a neuron output would rarely locate in cornercase region as well. Nevertheless, it does not mean that testing thecorner cases of a neuron is not important because defects of DNNs
could also locate in the corner-case regions (see Section 4.3).
To cover these corner-case regions of DNNs, we define twocoverage criteria, i.e., neuron boundary coverage and strong neuron activation coverage. Given a test input x, if ϕ(x, n) belongs to(−∞, lown ) or (highn , +∞), we say the corresponding corner-caseregion is covered. To quantify this, we first define the number ofcovered corner-case regions as follows:
UpperCornerNeuron = {n ∈ N | ∃x ∈ T : ϕ(x, n) ∈ (highn , +∞)};
LowerCornerNeuron = {n ∈ N | ∃x ∈ T : ϕ(x, n) ∈ (−∞, lown )}.
(ii) Neuron Boundary Coverage. Neuron boundary coveragemeasures how many corner-case regions (w.r.t. both of the upperboundary and the lower boundary values) have been covered by
the given test input set T . It is defined as the ratio of the number ofcovered corner cases and the total number of corner cases (2 × |N |):Definition 3.1. For a test input x ∈ T , we say that a DNN islocated in its major function region given x iff ∀n ∈ N : ϕ(x, n) ∈[lown , highn ].

NBCov(T ) =Some recent research on DNNs interpretability empirically showsthat the hyperactive neurons might potentially deliver useful learning patterns within DNNs [30, 61]. Based on this intuition, theproposed coverage criteria in the rest of this section focus more on
the hyperactive neuron cases (e.g., top-k neuron coverage in thenext subsection). Similar to neuron boundary coverage, we further
define strong neuron activation coverage to measure the coverage
status of upper-corner cases.
(iii) Strong Neuron Activation Coverage. Strong neuron activation coverage measures how many corner cases (w.r.t. the upperboundary value highn ) have been covered by the given test inputs
T . It is defined as the ratio of the number of covered upper-cornercases and the total number of corner cases (|N |):To exhaustively cover the major function regions, we partition[lown , highn ] into k sections, and require each of them to be covered by the test inputs. We name this coverage as k-multisection
neuron coverage.
(i) k-multisection Neuron Coverage. Given a neuron n, the kmultisection neuron coverage measures how thoroughly the given
set of test inputs T covers the range [lown , highn ]. To quantify
this, we divide the range [lown , highn ] into k equal sections (i.e.,k-multisections), for k > 0. We write Sin to denote the set of values
in the i-th section for 1 ≤ i ≤ k.
If ϕ(x, n) ∈ Sin , we say the i-th section is covered by the test
input x. Therefore, for a given set of test inputs T and the neuronn, its k-multisection neuron coverage is defined as the ratio of thenumber of sections covered by T and the total number of sections,
i.e., k in our definition. We define the k-multisection coverage of aneuron n as:|{Sin | ∃x ∈ T : ϕ(x, n) ∈ Sin }|k|UpperCornerNeuron| + |LowerCornerNeuron|.
2 × |N |SNACov(T ) =3.2.

|UpperCornerNeuron|.
|N |Layer-Level Coverage Criteria

At layer-level, we use the top hyperactive neurons and their combinations (or the sequences) to characterize the behaviors of a DNN.
For a given test input x and neurons n 1 and n 2 on the same layer,
we say n 1 is more active than n 2 given x if ϕ(x, n 1 ) > ϕ(x, n 2 ).

4 This paper focuses on feedforward neural networks. For recurrent neural networks (RNNs), we canunroll a certain depth of layers of an RNN and adapt ϕ(x, n) by setting x to be an input sequence.

123DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
For the i-th layer, we use topk (x, i) to denote the neurons thathave the largest k outputs on that layer given x. For example, in
Figure 1(a), assume ϕ(x, n 1 ) and ϕ(x, n 3 ) are larger than ϕ(x, n 2 ),the top-2 neurons on layer 1 are n 1 and n 3 (depicted in green).
(i) Top-k Neuron Coverage. The top-k neuron coverage measures
how many neurons have once been the most active k neurons oneach layer. It is defined as the ratio of the total number of top-k
neurons on each layer and the total number of neurons in a DNN:ÐÐ| x∈T ( 1≤i ≤l topk (x, i))|TKNCov(T , k) =.
|N |The neurons from the same layer of a DNN often play similarroles and the top active neurons from different layers are important indicators to characterize the major functionality of a DNN.
Intuitively, to more thoroughly test a DNN, a test dataset should
uncover more top active neurons.
(ii) Top-k Neuron Patterns. Given a test input x, the sequence ofthe top-k neurons on each layer also forms a pattern. In Figure 1(a),assume the neurons in green are the top-2 neurons on each layer,
the pattern can be represented as ({n 1 , n 3 }, {n 5 , n 6 }, {n 8 , n 9 }). Moreformally, a pattern is an element of 2L1 × 2L2 × · · · × 2Ll , where
2Li is the set of subsets of the neurons on i-th layer, for 1 ≤ i ≤ l.
Given the test input set T , the number of top-k neuron patterns for
T is defined as:

CW [8]) for comparative study. Each of the adversarial techniques
generates tests to detect DNN’s potential defects through the minor
perturbations on a given input, described as follows:• FGSM crafts adversarial examples using loss function J (Θ, x, y)with respect to the input feature vector, where Θ denotes themodel parameters, x is the input, and y is the output label of x, theadversarial example is generated as: x∗ = x+ϵ sign(∇x J (Θ, x, y)).
• BIM applies adversarial noise η many times iteratively with asmall parameter ϵ, rather than one η with one ϵ at a time, which∗ +gives a recursive formula: x∗0 = x and xi∗ = clipx,ϵ (xi−1∗ϵ sign(∇xi∗−1 J (Θ, xi−1 , y))), where clipx,ϵ (·) denotes a clippingof the values of the adversarial sample such that they are within
an ϵ-neighborhood of the original input x.
• JSMA is proposed for targeted misclassification. For an input x
and a neural network F , the output of class j is denoted as F j (x).
To achieve a target misclassification class t, Ft (x) is increased
while the probabilities F j (x) of all other classes j , t decrease,
until t = arg maxj F j (x).
• Carlini/Wagner (CW): Carlini and Wagner recently proposed
new optimization-based attack technique which is arguably themost effective in terms of the adversarial success rates achieved
with minimal perturbation [8]. In principle, the CW attack is toapproximate the solution to the following optimization problem:arg min

TKNPat(T , k) = |{(topk (x, 1), . . . , topk (x, l)) | x ∈ T }|.

where L is a loss function to measure the distance between theprediction and the ground truth, and the constant λ is to balance
the two loss contributions. In this paper, we adopt the CW∞ ,where each pixel is allowed to be changed by up to a limit.

EXPERIMENTS

We implement DeepGauge on Keras 2.1.3 [10] with TensorFlow 1.5.0
backend [1], and apply the proposed testing criteria to DNNs forevaluation in this section.

4.1

λL(x, x∗ ) − J (Θ, x∗ , y),x∗Intuitively, the top-k neuron patterns denote different kinds ofactivated scenarios from the top hyperactive neurons of each layer.

4

ASE ’18, September 3–7, 2018, Montpellier, France

Evaluation Subjects

Datasets and DNN Models. We select two popular publicly-available
datasets, i.e., MNIST [32] and ImageNet [42] (see Table 1) for evaluation. MNIST is for handwritten digits recognition, containing
70,000 input data in total, of which 60,000 are training data and10,000 are test data. On MNIST, we use three pre-trained LeNet
family models (LeNet-1, LeNet-4, and LeNet-5) [32] for analysis.
To further demonstrate the usefulness of our criteria towards
larger scale real-world DL systems, we also select ImageNet, a large
set of general image dataset (i.e., ILSVRC-2012 [42]) for classification
containing more than 1.4 million training data and 50,000 test datafrom 1,000 categories. The DNNs we used for ImageNet are pretrained VGG-19 [44] and ResNet-50 [24] models, both of which
are relatively large in size and obtain competitive records in the
ILSVRC competition [42], containing more than 16,000 and 94,000
neurons, and 25 and 176 layers, respectively. As a DNN testing
criterion towards future industry level application, we believe thescalability up-to ImageNet-like or even larger data size and model
size is almost indispensable.
Adversarial Test Input Generation. Besides using original test
data accompanied in the corresponding dataset for coverage evaluation, we further explore four state-of-the-art adversarial test input
generation techniques (i.e., FGSM [20], BIM [31], JSMA [37], and124

Figure 2 shows examples of the generated tests of the four adversarial techniques on the sampled data from MNIST test set. In
this example, we could see that compared with FGSM and BIM,
JSMA and CW perturb fewer pixels on the sampled test input. Furthermore, given the same input data but different DNNs, the same
technique would often generate different adversarial test results.
For example, given the input image 7, JSMA generates different results on DNNs (i.e., LeNet-1, LeNet-4 and LeNet-5). In other words,
the studied adversarial techniques are often DNN dependent.

4.2

Evaluation Setup

MNIST. On MNIST dataset, each image is single-channel of size
28 × 28 × 1. Before the evaluation starts, we first obtain the DNN’s
neuron output statistical information through runtime profilingeach of the studied DNNs (i.e., LeNet-1, LeNet-4, and LeNet-5) using
the 60,000 training data. When testing evaluation starts, for each
DNN under analysis, we run the 10,000 test data on the model
to obtain the corresponding coverage. For each studied DNN, we
further generate another four sets of adversarial test data which can
explore defects of DL,5 through FGSM [20], BIM [31], JSMA [37],and CW [8]. We show that DeepGauge is general and easy to betested on the state-of-the-art adversarial test generation techniques.
After generating the four adversarial datasets, we aggregate eachof them with the original MNIST test dataset (with a total size 20,000
for each), which enables us to perform the comparative study on
5 Each generated adversarial dataset is of the same size as the original test set.

ASE ’18, September 3–7, 2018, Montpellier, France

FGSM
FGSM

Test org.

BIM

JSMA
JSMA

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wang

CW
CW

FGSM

JSMA
JSMA

BIM

LeNet-1

Test org.

CW
CW

FGSM

BIM

LeNet-4

LeNet-1

JSMA
JSMA

CW
CW

LeNet-5

LeNet-4

LeNet-5

Figure 2: Examples of original sampled test data from MNIST in comparison to the ones generated by each adversarial technique on the corresponding studied DNN models.
Table 1: Breakdowns of datasets and DNN models.
DataSetDatasetDescription

MNIST

Digit recog.

ImageNet

General image
with 1000-classes

DNN Model#Neuron#Layer

LeNet-1
LeNet-4
LeNet-5
VGG-19ResNet-50

52
148
268
16,16894,059

7
8
9
25
176

Table 2: The parameter configurations for evaluation.

Test Data Source
for Eval.
Test org.&DL Coverage Criteria

k -multisection Neuron Cov. (KMNC)
Neuron Boundary Cov. (NBC)

FGSM/BIM/JSMA/CW

Strong Neuron Activation Cov. (SNAC)
Top-k Neuron Cov. (TKNC)
Top-k Neuron Patterns (TKNP)Test org.&
FGSM/BIM/CW

k=1,000
LB=l
UB=u
UB=u
k=1
k=1

Parameter Configuration
k=10,000
N.A.
LB=l − 0.5 ∗ σ
LB=l − σUB=u + 0.5 ∗ σ
UB=u + σ
UB=u + 0.5 ∗ σ
UB=u + σ
k=2
k=3
k=2
k=3

each of its labeled categories in the original ImageNet test dataset,with a total number of 5,000 images as the test data for our evaluation. We also try to use FGSM, BIM, JSMA, and CW to generate
adversarial tests for each of the studied DNNs. However, we are unable to set up JSMA to run successfully on either of the two DNNs.6
Overall, we have a total of 2 (models)×4 (datasets)×14 (criterion
settings) = 112 experimental configurations for ImageNet.
To support such large scale evaluation, we run the experiments
on a computer cluster. Each cluster node runs a GNU/Linux systemwith Linux kernel 3.10.0 on a 18-core 2.3GHz Xeon 64-bit CPU with
196 GB of RAM and also an NVIDIA Tesla M40 GPU with 24G.

how the adversarial test data enhances the defect detection ability
from our coverage criteria measurement. Since the studied adversarial test generation techniques are model dependent, the adversarial
datasets generated by the same adversarial techniques are actually different for each model, though the same number (i.e., five)datasets are used to evaluate on each model. For each adversarial
technique, we actually use it to generate three adversarial datasets,
one for each of LeNet-1, LeNet-4, and LeNet-5, respectively.
The detailed parameter configurations for each criterion are
shown in Table 2, where u and l are the output upper bound (maximal value) and lower bound (minimal value) obtained for eachneuron during profiling, respectively; σ is the standard deviation of
the outputs of a neuron during profiling. Although the definition ofneuron boundary coverage and strong neuron activation coverage
are based on u and l alone (i.e., neuron output UpperBound (UB) andLowerBound (LB)), it would also be interesting to see what results
could be obtained if we further tighten corner-case regions (i.e.,
increase upper bound and decrease lower bound). Therefore, besides setting UB and LB to u and l as defined in Section 3.1, we
also evaluate another two configurations by increasing UB (resp.
decreasing LB) by 0.5 ∗ σ and σ for neuron boundary coverage andstrong neuron activation coverage (see Table 2). In total, we have
3 (models)×5 (datasets)×14 (criterion settings) = 210 evaluation
configurations for MNIST.
ImageNet. ImageNet (ILSVRC-2012) [42] is more challenging forevaluation due to its large data size (more than 1.4 million trainingdata) as well as large image size (224 × 224 × 3) for processing.
Moreover, the DNNs that achieve high accuracy are often complex. Compared with LeNet family models, the studied VGG-19 andResNet-50 are much more complex in terms of both neurons andlayers. Due to the computational complexity of adversarial test generation on ImageNet for analysis, we randomly sample images from

4.3

Experimental Results

In our experiments, we have seen useful testing feedbacks frommultiple perspectives with each testing criterion, showing someunique portrayal of the runtime behavior of DNNs. We first describe
some obtained results and then summarize our findings.7
4.3.1 MNIST and ImageNet. MNIST. As shown in Table 3, thecoverage of different criteria obtained by the adversarial techniques
generally increase compared with the original MNIST test dataset.
For instance, as for LeNet-4, the JSMA increases the coverage of theoriginal tests from 39.7% to 52.3% by 31.7% in 10, 000-multisection
neuron coverage, from 9.1% to 16.2% by 78% in neuron boundary
coverage, from 13.5% to 27.7% by 105% in strong neuron activation
coverage, from 62.2 to 66.2 by 6.6% in top-1 neuron coverage, andfrom 787 to 1,395 by 77.3% in top-1 neuron patterns.
The increase of coverage infers that the adversarial test data
overall explore new DNNs’ internal states, some of which are not
6 This could be potentially caused by the massive data size and the complexity of VGG-19 and ResNet-

50. The similar issue on JSMA was also reported in a previous work [57].

7 Due to the page limit, we put more detailed experimental result discussion, as well as data plot on

the paper’s accompanying website https://deepgauge.github.io/.

125

DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

Table 4: Coverage results on ImageNet, VGG-19 and ResNet50, and generated tests by adversarial techniques.

Table 3: Coverage results of DeepGauge on MNIST, LetNetmodels, and generated tests by adversarial techniques.
Testing
Criteria

DNN
LN-1

KMNC
(%)

LN-4
LN-5
LN-1

NBC
(%)

LN-4

LN-5

LN-1

SNAC
(%)

LN-4

LN-5

LN-1

TKNC
(%)

LN-4

LN-5

LN-1

TKNP

LN-4

LN-5

Eval. Config.

Test org.

k=1,000
k=10,000
k=1,000
k=10,000
k=1,000
k=10,000
LB=l ,UB=ul -0.5*σ ,u +0.5*σ
l -σ ,u +σ
LB=l ,UB=ul -0.5*σ ,u +0.5*σ
l -σ ,u +σ
LB=l ,UB=ul -0.5*σ ,u +0.5*σ
l -σ ,u +σ
UB=u
UB=u +0.5*σ
UB=u +σ
UB=u
UB=u +0.5*σ
UB=u +σ
UB=u
UB=u +0.5*σ
UB=u +σ
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3

64.5
37.3
70.4
39.7
68.5
37.2
43.3
17.3
8.7
9.1
0.7
0.3
8.6
1.7
1.1
38.5
23.1
17.3
13.5
1.4
0.7
14.9
3.4
2.2
61.5
86.5
92.3
62.2
79.1
85.1
49.3
63.8
72.4
76
915
3,716
787
6,190
9,301
1,136
6,947
9,684

Test org.
+FGSM
74.8
48.6
75.5
49.7
72.0
46.0
47.1
21.2
8.7
12.2
1.0
0.3
10.5
2.1
1.3
42.3
25.0
17.3
16.2
1.4
0.7
16.4
4.1
2.6
61.5
88.5
92.3
63.5
79.7
87.2
53.7
66.8
73.9
77
1,271
6,069
1,210
11,185
18,5152,141
13,987
19,361

Test org.
+BIM
68.1
46.6
73.6
50.2
71.5
47.6
49.0
21.2
8.7
13.9
1.0
0.3
11.6
1.9
1.3
46.2
28.9
17.3
18.9
1.4
0.7
20.2
3.7
2.6
61.5
88.5
92.3
64.9
80.4
87.8
53.7
67.5
74.6
77
1,185
5,708
1,140
11,26818,491
1,775
12,614
19,215

Test org.
+JSMA
77.7
51.5
77.7
52.3
73.8
48.8
46.2
18.3
9.6
16.2
2.7
1.0
13.4
3.0
1.7
42.3
25.0
19.2
27.7
5.4
2.0
23.5
6.0
3.4
63.5
88.5
92.3
66.2
81.8
86.5
51.9
66.0
74.3
100
1,325
6,597
1,395
12,14018,356
2,031
12,797
19,201

ASE ’18, September 3–7, 2018, Montpellier, France

Testing
Criteria

Test org.
+CW
72.8
49.5
74.5
50.1
71.2
46.8
44.2
17.3
8.7
10.5
1.0
0.3
9.1
2.0
1.1
40.4
23.1
17.3
13.5
1.4
0.7
14.9
3.4
2.2
61.5
86.5
92.3
64.9
80.4
86.5
52.2
66.0
74.6
86
1,270
5,823
1,389
11,742
18,1942,011
12,45619,157

KMNC
(%)

DNN
VGG-19ResNet-50
VGG-19NBC(%)
ResNet-50

VGG-19SNAC(%)
ResNet-50

VGG-19TKNC(%)
ResNet-50

VGG-19TKNP
ResNet-50

Eval. Config.

Test org.

k=1,000
k=10,000
k=1,000
k=10,000
LB=l ,UB=ul -0.5*σ , u +0.5*σ
l -σ , u +σ
LB=l ,UB=ul -0.5*σ , u +0.5*σ
l -σ , u +σ
UB=u
UB=u +0.5*σ
UB=u +σ
UB=u
UB=u +0.5*σ
UB=u +σ
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3
k=1
k=2
k=3

32.2
13.5
43.0
22.8
2.8
1.5
1.1
4.1
1.5
0.9
4.6
3.0
2.1
4.7
2.1
1.3
58.8
74.3
81.6
26.8
36.0
42.3
4,999
4,999
4,999
4,999
4,999
4,999

Test org.
+FGSM
36.9
18.8
47.5
29.3
8.7
4.0
3.2
6.9
2.1
1.2
10.5
8.0
6.3
7.0
2.8
1.8
61.5
76.2
82.9
30.3
38.6
44.7
8,265
9,581
9,921
9,998
9,998
9,998

Test org.
+BIM
38.0
19.1
47.8
29.6
7.4
3.4
2.5
11.5
6.0
3.9
9.8
6.8
5.1
13.1
8.3
6.1
68.1
80.8
85.9
29.1
38.3
44.3
9,989
9,998
9,998
9,998
9,998
9,998

Test org.
+CW
35.7
18.5
47.4
29.4
2.9
1.5
1.1
4.7
1.7
0.9
4.7
3.1
2.2
5.4
2.4
1.4
68.7
81.2
85.9
29.5
38.3
44.3
9,816
9,816
9,816
9,948
9,948
9,948

The top-1 neuron coverage obtained by both MNIST and ImageNet (see Table 3 (TKNC) and Table 4 (TKNC)) show that manyneurons of a DNN have been triggered into a top-k (i.e., 1, 2, and 3
in our evaluated cases) hyperactive states. For example, on VGG-19,
the sampled original tests of ImageNet achieve 58.8% top-1 neuron
coverage, and 81.6% top-3 neuron coverage. Although the top-k
coverage improvement is not that obvious compared with othercriteria, the adversarial data still trigger more neurons as top-k
activated neurons in many cases, which detects the hidden defects.
For different test input datasets, it is often the case that only afixed subset of neurons of each layer would function as top hyperactivated neurons. This would be a hint that the top hyperactivatedneurons of each layer might describe the high-level major function
skeleton of a neuron network. In comparison with the top-k neuron
patterns (see Table 3 (TKNP) and Table 4 (TKNP)), albeit most of
the top hyperactive neurons are relatively stable for each layer,
their combination still captures the structural difference of input
data.8 These two layer-level criteria altogether provide us with theinformation on which neurons matter the most within each layer;
and the top-k neuron patterns would mostly be able to differentiate
the input data when k is properly selected given a target DNN. The
findings indicate that generating tests to cover more top-k neuron
patterns would have a higher chance to find defects of a DNN.

covered by the original tests. As such adversarial test data reveal
defects of studied LeNet models, it indicates that generating tests
towards improving the coverage of the proposed criteria might potentially trigger more states of a DNN, incurring higher chances ofdefect detection, which is consistent with the usage of test coveragein traditional software testing.
ImageNet. The testing coverage (Table 4) on the ImageNet sharessome similarity with MNIST data while showing some differences.
VGG-19 and ResNet-50 models are much larger in size and complexity, potentially causing the obtained coverage lower than thatof LeNet in many cases. Consider the 10, 000-multisection neuron
coverage, FGSM achieves 48.6% on LeNet-1, but only 18.8% on VGG19. At first glance, it is tempting to draw the conclusion that a
DNN with higher complexity in terms of number of neurons andlayers is more difficult to be covered by tests. Our results show that
this might not be generally applicable. For example, the original
tests achieves 22.8% KMNC (k = 10, 000) on ResNet-50, but only
obtains 13.5% on VGG-19, although ResNet-50 has a larger numberof neurons and layers. Compared with MNIST, the generated adversarial tests on ImageNet incur even higher increase on the neuron
boundary coverage (NBC) and strong neuron activation coverage(SNAC) (Table 4). For example, on ResNet50 under LB=l and UB=u
configuration, BIM increases these two criteria by 280% (from 4.1%
to 11.5%) and 279% (from 4.7% to 13.1%), respectively.

4.3.2 Findings and Remarks. The overall experimental results demonstrate the usefulness of our proposed testing criteria for DNNs andare also helpful to explain the difference of the state-of-the-practiceadversarial techniques form multiple perspectives:• The original test data of MNIST and ImageNet cover both the
DNNs major function region (see KMNC) as well as corner-caseregion (see NBC and SNAC) in Tables 3 and 4. This also happens
8 Our in-depth investigation on the generated top-k neuron patterns for ImageNet show that therelatively large pattern coverage improvement (even for the top-1 case) is relevant to the large #of neurons and layers in VGG-19 and ResNet-50.

126

ASE ’18, September 3–7, 2018, Montpellier, France

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wang

to the generated adversarial datasets, showing that a defect of
DNN can occur either in a major function region or a corner-caseregion, both of which should be extensively tested.
• The test data generated by four studied adversarial techniques(combined with original test data) generally boost the coverage ofour criteria. Since an adversarial test data could potentially reveal
defects of a DL system, it means that boosting the coverage of our
testing criteria could to some extent enhance the fault detection
ability, which is consistent with the practical purpose of testing
criteria widely adopted in traditional software testing. It also
shows that our test criteria metrics could capture the DNNs’internal behavioral difference of benign and adversarial test data.
We note that increasing the test coverage does not necessarily
imply that new defects could be detected in traditional software
testing. The same conclusion applies to our coverage criteria for
DNNs as well, though better-defined coverage criteria would bemuch more pronounced in finding defects.
• Test data (including the generated test data by adversarial) evaluated on both MNIST and ImageNet mostly obtain a higher
k-multisection neuron coverage than the neuron boundary coverage and strong neuron activation coverage, revealing that thetest data cover more of the major function region than the cornercase region of a DNN. The design of future DL testing techniquesshould also take account of covering corner-case regions.
• For most of the evaluated configurations, we find that a higher
strong neuron activation coverage is more achieved than its
corresponding neuron boundary coverage. This might be causedby the unique characteristics of those activation functions in ourstudied DNNs, which makes the lower region (small value) moredifficult to be covered than the upper region of the statisticalprofiling distribution.9 This observation is consistent with themodels we studied, as LeNet family, VGG-19, and ResNet-50 all
use ReLU as activation functions, which could make the lower
regions of a neuron much smaller than the upper regions.10

Test orig.
100.0%

TestTest
orig.
TestTest
orig.+FGSM
Test orig.+BIM
orig.+BIM
orig.+JSMA
Test orig.+CW

Test orig.+JSMA

Test orig.+FGSM

100% 100% 100% 100% 100%

100.0%

100% 100% 100% 100% 100%

Test orig.+CW

99.3% 99.6% 99.3% 99.3% 99.3%

90.0%

85.8% 85.1%
80.8% 80.8% 80.8%

78%

79.1%

81.8% 83.1%

85.1%

86.9% 86.2% 86.2% 85.5%

73.1%

80.0%

72.0%

80.0%
60.0%

54.0%

60.0%

40.0%

36.0%

18.0%

20.0%

40.0%
0.0%

0.0%

LeNet-1
Test
orig.+FGSM

Test orig.

20.0%

LeNet-4
Test orig.+BIM

LeNet-5 Test orig.+CW
Test orig.+JSMA

LeNet-1

LeNet-4

LeNet-5(2) Threshold=0.2(1) Threshold=0

80.0%

70.0%

66.2% 66.2% 66.9% 66.2% 66.9%

73.5% 73.9% 73.5% 73.9% 73.5%

65.3% 66%67.2%

65.7% 65.7%

68.2% 68.2% 68.2% 68.2% 68.2%

64.0%

56.0%

0.0%

LeNet-1

LeNet-4
42.0%(1) Threshold=0

48.0%

LeNet-5

38.5% 38.5% 38.5% 38.5% 38.5%

38.5% 38.5% 38.5% 38.5% 38.5%

32.0%

28.0%

16.0%

14.0%

0.0%

0.0%

LeNet-1

LeNet-4(3) Threshold=0.5

LeNet-5

LeNet-1

LeNet-4(4) Threshold=0.75

LeNet-5

Figure 3: The DeepXplore neuron coverage results on MNIST
dataset under different threshold configurations.

4.4

Comparison with DeepXplore’s Neuron
Coverage (DNC)

Pei et al. [38] propose a kind of neuron activation coverage as themeasurement for testing data diversity of a DNN and argue that thehigher the activation coverage, the more states of a DNN could be
explored, with a higher chance for defects detection. A key parameter of DNC is a user-specified threshold, and if an output of a neuron
is larger than the threshold, the neuron is counted as covered. To
demonstrate the difference between our set of criteria and DNC,
we set up the DNC evaluation with the same dataset, model, as wellas adversarial data generation settings as described in Section 4.1.
For the threshold parameter, we first set thresholds to be 0 and 0.75,
as used in [38]; to make an even more comprehensive comparison,
we also use two other settings (i.e., 0.2 and 0.5). Figures 3 and 4
show that the results of DNC obtained on the original test dataset
and the dataset generated by adversarial techniques for MNISTand ImageNet are almost the same for all experimental settings,
indicating that DNC is unable to differentiate the original test datafrom adversarially generated ones, which trigger the correct andincorrect behaviors of a DNN, respectively. This means that DNC
could hardly capture the difference between original test data andcorresponding test data generated by adversarial techniques. However, to detect the defects of DNNs in a more fine-grained level, it isnecessary that the coverage criteria capture such minor differences,where the defects (adversarially triggered states) also lie in.
Our further in-depth investigation on DNC reveals that, this coverage criterion imposes several limitations: (1) DNC uses the same
threshold as the activation evaluation for all the neurons. However,
we find that the output statistical distribution of different neurons
are quite diverse. Given a test suite for analysis, the outputs ofsome neurons may exhibit quite a small variance with a large meanvalue, while others might have a large variance with a low meanvalue. Therefore, using the same threshold for all neurons without considering the disparity in neuron’s functional distributions
would greatly diminish the accuracy. For example, given a neuron with very small mean and standard deviation, even a slightly
larger user-specified threshold would generally determine that thisneuron cannot be covered. (2) DNC normalizes the dynamic range

Remark 1. In general, for neuron boundary coverage andstrong neuron activation coverage, the higher (resp. lower)
the neuron’s upper (resp. lower) bound, the less increment on
coverage we observe; for top-k neuron coverage, the larger thevalue of k, the less increment on coverage; for top-k neuron
patterns, the larger the value of k, the more increment on
patterns.
Remark 2. For neuron boundary coverage and strong neuron activation coverage, the 4 adversarial techniques havesufficient diversity on the performance, which is similar to
traditional test generation which aim to cover different potential defects. Specifically, we observe that the adversarial tests
generated by CW are harder to be distinguished by the test
coverage since the CW perturbation concentrates more on
the objects with smaller magnitude, which may trigger lessinternal behavior changes of DNNs.

9 LeNet-1 is the only exceptional case, which might be caused by the over-simplicity of its network.
10 In particular, ReLU function propagates the positive output of a neuron to the next layer while

blocking the negative output by setting it to zero, which stops influencing its following layers.

127

DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
Test orig.
100%

Test orig.+FGSM
99.92% 99.92% 99.92% 99.92%

Test orig.+BIM

Test orig.+CW Test orig.

Test orig.+FGSM

ASE ’18, September 3–7, 2018, Montpellier, France

Test orig.
Test orig.+BIM

70.6% 71.1% 71.3%

68.5% 68.9% 69.4%

64%

69%

Test orig.+BIM
60%

100%
72%

80%

Test
orig.+FGSM
Test
orig.+CW

87.9% 88.7% 88.6% 88.4% 80%

99.99% 99.99% 99.99% 99.99% 90%

Test orig.+CW

56.5% 56.7% 56.7% 56.7%

71%

48%
57.2% 57.2% 57.2% 57.2%

80%
60%

54%

40%

36%

20%

18%

60%

48%

36%

32%

24%

16%

12%

8.6%

9.3%

9.4%

9.1%

40%
0%

0%

0%

VGG19

ResNet50(1) Threshold=0

VGG19
20%

0%

VGG19

ResNet50

ResNet50(3) Threshold=0.5(2) Threshold=0.2

VGG19

ResNet50(4) Threshold=0.75

Figure 4: The DeepXplore neuron activation coverage results on ImageNet dataset for different threshold settings.
0%

VGG19

ResNet50

of neuron outputs according to max and min output of neurons
5 RELATED WORK
(1) Threshold=0
on the corresponding layer for each input image under analysis.
In this section, we attempt to review the most relevant work inThis raises an issue that the same normalized activation value (e.g.,
three aspects: testing, verification, and security of DL systems.
0.3) means differently for different input data, as the max and min
output of each layer might change for each input, which renders
5.1 Testing of DL Systems
the notion of “larger than a given threshold” inconsistent among
Traditional practices in measuring machine learning systems mainly
different inputs.
rely on probing their accuracy on test inputs which are randomly
Doing so will also eliminate the relativity in activation magnitude
drawn from manually labeled datasets and ad hoc simulations [54].
among different inputs, which is a very important property to
However, such black-box testing methodology may not be able tosupport the findings of neuron activation coverage. We instead
find various kinds of corner-case behaviors that may induce unspecify the upper and lower bounds for each neuron obtained fromexpected errors [19]. Wicker et al. [53] recently proposed a Scalethe analysis on the training dataset, as opposed to each individual
Invariant Feature Transform feature guided black-box testing andinput. In other words, our method relies on the statistics of theshowed its competitiveness with CW and JSMA along this direction.
training set which is used to determine the main functionality of
Pei et al. [38] proposed a white-box differential testing algorithm
the DNN system.
for systematically finding inputs that can trigger inconsistencies
between multiple DNNs. They introduced neuron coverage for
4.5 Threats to Validity and Discussion
measuring how much of the internal logic of a DNN has been
The selection of evaluation subjects (i.e., dataset and DNN modtested.However, it still exhibits several caveats as discussed in Secels) could be a threat to validity. We try to counter this by using
tion 4.4. DeepTest [49] investigates a basic set of image transformathe commonly-studied MNIST dataset and the practical large-scaletions (e.g., scaling, shearing, and rotation) from OpenCV and shows
dataset ImageNet; for each studied dataset, we use the well-known
that they are useful to detect defects in DNN-driven autonomous
pre-trained models of different sizes and complexity ranging fromcars. Along this direction, DeepRoad [59] uses input image scene
52 neurons up to more than 90,000 neurons. Even though, sometransformation and shows its potentiality with two scenes (i.e.,of our results might not generalize to other datasets and DNN
snowy and rainy) for autonomous driving testing. The scene transmodels. Another threat could be caused by the configurable hyperformation is obtained through training a generative adversarial
parameters in the coverage criteria definition. As a countermeasure
network (GAN) with a pair of collected training data that cover thewhile considering the limited computational resources, we evaluate
statistical features of the two target scenes.
each criterion with different settings, and analyze the influence ofCompared with traditional software, the dimension and potential
the parameters on criteria accuracy. Even though, it might still nottesting space of a DNN is often quite large. DeepCT [35] adapts thecover the best parameter use-cases. For example, our evaluation
concept of combinatorial testing, and proposes a set of coveragestudied k = 1, 000 and k = 10, 000 for k-multisection neuron coverbased on the neuron input interaction for each layer of DNNs, toage. We leave the optimized hyper-parameter selection in our future
guide test generation towards achieving reasonable defect detecwork. Further threat could be caused by the quality of training datation ability with a relatively small number of tests. Inspired byused for distribution (i.e., the interval range) analysis of neuron
the MC/DC test criteria in traditional software [29], Sun et al. [47]output. In this paper, we consider publicly available well-pretrained
proposed a set of adapted MC/DC test criteria for DNNs, and show
DNN models accompanied by training data with good quality.
that generating tests guided by the proposed criteria on small scale
For adversarial test generation, we select four popular state-ofneural networks (consisting of Dense layers with no more than 5
the-practice techniques to simulate defects from different sources
hidden layers and 400 neurons) exhibits higher defect detection
and granularity. We either follow the authors’ suggested settings orability than random testing. However, whether MC/DC criteria
use their default settings. Moreover, to make comprehensive comscale to real-world-sized DL systems still needs further investigaparisons with DeepXplore’s neuron coverage (DNC), we evaluatetion. Instead of observing the runtime internal behaviors of DNNs,
DNC with multiple threshold settings.
DeepMutation [34] proposes to mutate DNNs (i.e., injecting faults

128

ASE ’18, September 3–7, 2018, Montpellier, France

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wang

either from the source level or model level) to evaluate the test data
quality, which could potentially be useful for test data prioritizationin respect of robustness on a given DNN.
Our work of proposing multi-granularity testing coverage for DL
systems is mostly orthogonal to the existing work. Compared withthe extensive study on traditional software testing, testing DL is
still at an early stage. Most existing work on DL testing lacks somesuitable criteria to understand and guide the test generation process.
Since test generation guided by coverage criteria (e.g., statement
coverage, branch coverage) towards the exploration of diverse software states for defect detection has become the de facto standard
in traditional software testing [5, 16, 17, 33], the study to design
suitable testing criteria for DL is desperately demanding. This paper
makes an early attempt towards this direction by proposing a set of
testing criteria. Our criteria not only can differentiate state-of-theart adversarial test generation techniques, but also potentially be
useful for the measurement of test suite diversity by analyzing the
DNNs’ internal states from multiple portrayals. We believe that our
proposed criteria set up an important cornerstone and bring a newopportunity to design more effective automated testing techniques
guided by testing criteria for DL systems.

5.2

With the rapid development of adversarial attack techniques,
extensive studies have been performed to circumvent adversarial
attacks. Galloway et al. [18] recently observe that low-precision
DNNs exhibit improved robustness against some adversarial attacks. This is primarily due to the stochastic quantization in neuralnetwork weights. Ensemble adversarial training [51], GAN based
approaches [43, 45], random resizing and random padding [56],game theory [13], and differentiable certificate [41] methods are all
investigated to defend against adversarial examples. By applying
image transformations, such as total variance minimization andimage quilting, very effective defenses can be achieved when the
network is trained on the aforementioned transformed images [23].
For more extensive discussion on current state-of-the-art defense
techniques, we refer readers to [4].
Our proposed testing criteria enable the quantitative measurement of different adversarial attack techniques from the software
engineering perspective. This could be potentially helpful for understanding and interpreting DNNs’ behaviors, based on whichmore effective DNN defense technique could be designed. In future
work, it would be also interesting to examine how to integrate theproposed testing criteria into the DL development life cycle towards
building high quality DL systems.

Verification of DL Systems
6

Formal methods can provide formal guarantees about safety androbustness of verified DL systems [22, 28, 39, 40, 50, 53]. The mainconcern of formal methods are their scalability for real-world-sized(e.g., 100, 000 neurons or even more) DL systems.
The early work in [40] provided an abstraction-refinement approach to checking safety properties of multi-layer perceptrons.
Their approach has been applied to verify a network with only 6neurons. DLV [53] can verify local robustness of DL systems w.r.t.
a set of user specified manipulations. Reluplex [28] is a sound andcomplete SMT-based approach to verifying safety and robustness of
DL systems with ReLU activation functions. The networks verified
by Reluplex in [28] have 8 layers and 300 ReLU nodes. DeepSafe[22] uses Reluplex as its verification engine and has the same scalability problem as Reluplex. AI2 [50] is a sound analyzer based on
abstract interpretation that can reason about safety and robustness
of DL systems. It trades precision for scalability and scales better
than Reluplex. The precision of AI2 depends on abstract domains
used in the verification, and it might fail to prove a property whenit actually holds. VERIVIS [39] can verify safety properties of DL
systems when attackers are constrained to modify the inputs only
through given transformation functions. However, real-world transformations can be much more complex than the transformation
functions considered in the paper.

5.3

CONCLUSION AND FUTURE WORKThe wide adoption of DL systems, especially in many safety-criticalareas, has posed a severe threat to its quality and generalization
property. To effectively measure the testing adequacy and lay down
the foundation to design effective DL testing techniques, we propose a set of testing criteria for DNNs. Our experiments on two
well-known datasets, five DNNs with diverse complexity, and fourstate-of-the-art adversarial testing techniques show that the tests
generated by the adversarial techniques incur obvious increasesof the coverage in terms of the metrics defined in the paper. This
demonstrates that DeepGauge could be a useful indicator for evaluating testing adequacy of DNNs.
To the best of our knowledge, our work is among the earlystudies to propose testing criteria for DL systems. We expect thatthe proposed testing criteria could be particularly amenable to DL
testing in the wild. In the next step, we will continue to explore
alternative testing criteria for DNNs, such as the combination ofboth hyperactive and hypoactive neurons. We also plan to study
the proposed testing criteria guided automated test generation
techniques for DNNs. We hope that our study not only provides anavenue to illuminate the nature and mechanism of DNNs, but alsolays the foundation towards understanding and building genericand robust DL systems.

Attacks and Defenses of DL Systems

ACKNOWLEDGMENTS

A plethora of research has shown that deep learning systems canbe fooled by applying carefully crafted adversarial perturbation
added to the original input [6–9, 20, 48, 55, 60], many of which are
based on gradient or optimization techniques. However, it still lacks
extensive study on how these adversarial techniques differentiate
in terms of DNNs’ internal states. In this study, we make an earlyattempt towards such a direction based on our proposed criteria.

This work was partially supported by National Key R&D Programof China 2017YFC1201200 and 2017YFC0907500, Fundamental Research Funds for Central Universities of China AUGA5710000816,
JSPS KAKENHI Grant 18H04097. We gratefully acknowledge thesupport of NVIDIA AI Tech Center (NVAITC) to our research. We
also appreciate the anonymous reviewers for their insightful andconstructive comments.

129

DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

REFERENCES

ASE ’18, September 3–7, 2018, Montpellier, France[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual
Learning for Image Recognition. In CVPR. 770–778.

[1] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey
Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray,
Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan
Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and
Implementation (OSDI 16). 265–283.

[25] Warren He, Bo Li, and Dawn Song. 2018. Decision Boundary Analysis of Adversarial Examples. In ICLR.
[26] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed,
Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N
Sainath, et al. 2012. Deep Neural Networks for Acoustic Modeling in Speech
Recognition: The Shared Views of Four Research Groups. IEEE Signal Processing
Magazine 29, 6 (2012), 82–97.

[2] Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing (1 ed.).
Cambridge University Press, New York, NY, USA.

[27] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of
Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (Sept. 2011), 649–678.

[3] Cyrille Artho, Quentin Gros, Guillaume Rousset, Kazuaki Banzai, Lei Ma, Takashi
Kitamura, Masami Hagiya, Yoshinori Tanabe, and Mitsuharu Yamamoto. 2017.
Model-Based API Testing of Apache ZooKeeper. In 2017 IEEE Int. Conf. on Software
Testing, Verification and Validation (ICST 2017). Tokyo, Japan, 288–298.

[28] Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer.
2017. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks.
International Conference on Computer Aided Verification (CAV) (2017).

[4] Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated Gradients
Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.
In Proceedings of the 35th International Conference on Machine Learning, ICML
2018. https://arxiv.org/abs/1802.00420[29] Hayhurst Kelly J., Veerhusen Dan S., Chilenski John J., and Rierson Leanna K.
2001. A Practical Tutorial on Modified Condition/Decision Coverage. Technical
Report.
[30] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda
Viegas, et al. 2018. Interpretability beyond Feature Attribution: Quantitative
Testing with Concept Activation Vectors (TCAV). In International Conference on
Machine Learning. 2673–2682.

[5] Benoit Baudry and Martin Monperrus. 2015. The Multiple Facets of Software
Diversity: Recent Developments in Year 2000 and Beyond. ACM Computing
Surveys (CSUR) 48, 1 (2015), 16.
[6] Wieland Brendel, Jonas Rauber, and Matthias Bethge. 2018. Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models.
In ICLR.

[31] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017. Adversarial Examples
in the Physical World. ICLR (2017).
[32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradientbased Learning Applied to Document Recognition. Proc. IEEE 86, 11 (1998),
2278–2324.

[7] Nicholas Carlini and David Wagner. 2017. Adversarial Examples are not Easily
Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM
Workshop on Artificial Intelligence and Security. ACM, 3–14.

[33] Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, andRudolf Ramler. 2015. GRT: Program-Analysis-Guided Random Testing (T). In
Proceedings of the 2015 30th IEEE/ACM International Conference on AutomatedSoftware Engineering (ASE) (ASE ’15). IEEE Computer Society, Washington, DC,
USA, 212–223. https://doi.org/10.1109/ASE.2015.49[8] Nicholas Carlini and David Wagner. 2017. Towards Evaluating the Robustness of
Neural Networks. In IEEE Symposium on Security and Privacy. 39–57.
[9] Sen Chen, Minhui Xue, Lingling Fan, Shuang Hao, Lihua Xu, Haojin Zhu, and
Bo Li. 2018. Automated Poisoning Attacks and Defenses in Malware Detection
Systems: An Adversarial Machine Learning Approach. Computers & Security 73(2018), 326–344.

[34] Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Fei Juefei-Xu, Chao
Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepMutation:
Mutation Testing of Deep Learning Systems. International Symposium on Software
Reliability Engineering (ISSRE) (2018).

[10] François Chollet et al. 2015. Keras. https://github.com/fchollet/keras.
[11] Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. 2012. Multi-column Deep
Neural Networks for Image Classification. In CVPR. 3642–3649.

[35] Lei Ma, Fuyuan Zhang, Minhui Xue, Bo Li, Yang Liu, Jianjun Zhao, and Yadong
Wang. 2018. Combinatorial Testing for Deep Learning Systems. arXiv preprint
arXiv:1806.07723 (2018).

[12] Dan Ciresan, Alessandro Giusti, Luca M Gambardella, and Jürgen Schmidhuber. 2012. Deep Neural Networks Segment Neuronal Membranes in Electron
Microscopy Images. In NIPS. 2843–2851.

[36] Glenford J. Myers, Corey Sandler, and Tom Badgett. 2011. The Art of Software
Testing (3rd ed.). Wiley Publishing.

[13] Guneet S. Dhillon, Kamyar Azizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi,
Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar. 2018. Stochastic
Activation Pruning for Robust Adversarial Defense. In ICLR.

[37] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik,and Ananthram Swami. 2016. The Limitations of Deep Learning in Adversarial
Settings. In Security and Privacy (EuroS&P), 2016 IEEE European Symposium on.
IEEE, 372–387.

[14] Arilo C. Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H. Travassos. 2007. A Survey on Model-based Testing Approaches: A Systematic Review.
In Proc. 1st ACM Int’l Workshop on Empirical Assessment of Software Engineering
Languages and Technologies. 31–36.

[38] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated Whitebox Testing of Deep Learning Systems. In Proceedings of the 26th
Symposium on Operating Systems Principles. 1–18.

[15] ECSS. 2009. Space engineering - Software.

[39] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Towards Practical
Verification of Machine Learning: The Case of Computer Vision Systems. CoRR
abs/1712.01785 (2017). arXiv:1712.01785 http://arxiv.org/abs/1712.01785[16] Robert Feldt, Richard Torkar, Tony Gorschek, and Wasif Afzal. 2008. Searchingfor Cognitively Diverse Tests: Towards Universal Test Diversity Metrics. In Proceedings of the 2008 IEEE International Conference on Software Testing Verification
and Validation Workshop. IEEE, 178–186.

[40] Luca Pulina and Armando Tacchella. 2010. An Abstraction-Refinement Approach
to Verification of Artificial Neural Networks. In International Conference on
Computer Aided Verification. Springer, 243–257.

[17] Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE
Trans. Softw. Eng. 39, 2 (Feb. 2013), 276–291. https://doi.org/10.1109/TSE.2012.14[41] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified Defenses
against Adversarial Examples. In ICLR.

[18] Angus Galloway, Graham W. Taylor, and Medhat Moussa. 2018. Attacking
Binarized Neural Networks. In ICLR.

[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C.
Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge.
IJCV 115, 3 (2015), 211–252.

[19] Ian Goodfellow and Nicolas Papernot. 2017. The Challenge of Verification andTesting of Machine Learning.
[20] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining andHarnessing Adversarial Examples. In ICLR.

[43] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. 2018. Defense-GAN:
Protecting Classifiers Against Adversarial Attacks Using Generative Models. In
ICLR.

[21] Google Accident. 2016.
A Google Self-driving Car Caused a Crashfor the First Time.
https://www.theverge.com/2016/2/29/11134344/
google-self-driving-car-crash-report[44] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-scale Image Recognition. ICLR (2015).

[22] Divya Gopinath, Guy Katz, Corina S. Pasareanu, and Clark Barrett. 2018. DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural
Networks. International Symposium on Automated Technology for Verification andAnalysis (ATVA) (2018).

[45] Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman.
2018. PixelDefend: Leveraging Generative Models to Understand and Defend
against Adversarial Examples. In ICLR.

[23] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. 2018.
Countering Adversarial Images using Input Transformations. In ICLR.

[46] Ting Su, Ke Wu, Weikai Miao, Geguang Pu, Jifeng He, Yuting Chen, and Zhendong
Su. 2017. A Survey on Data-Flow Testing. ACM Computing Surveys (CSUR) 50, 1,
Article 5 (March 2017), 35 pages.

130ASE ’18, September 3–7, 2018, Montpellier, France

Ma, Xu, Zhang, Sun, Xue, Li, Chen, Su, Li, Liu, Zhao, and Wang[47] Y. Sun, X. Huang, and D. Kroening. 2018. Testing Deep Neural Networks. ArXiv
e-prints (March 2018). arXiv:cs.LG/1803.04792[54] Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. 2016. Data Mining:
Practical Machine Learning Tools and Techniques. Morgan Kaufmann.

[48] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian Goodfellow, and Rob Fergus. 2014. Intriguing Properties of Neural Networks.
In ICLR.

[55] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song.
2018. Spatially Transformed Adversarial Examples. In ICLR.
[56] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. 2018.
Mitigating Adversarial Effects through Randomization. In ICLR.

[49] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: AutomatedTesting of Deep-Neural-Network-driven Autonomous Cars. In International Conference on Software Engineering (ICSE). ACM.

[57] Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature Squeezing: Detecting
Adversarial Examples in Deep Neural Networks. In Network and Distributed
System Security Symposium (NDSS).

[50] Dana Drachsler-Cohen Petar Tsankov Swarat Chaudhuri Martin Vechev Timon Gehr, Matthew Mirman. 2018. AI2: Safety and Robustness Certification of
Neural Networks with Abstract Interpretation. In IEEE Symposium on Security
and Privacy (SP).

[58] Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke.
2015. Towards Vision-based Deep Reinforcement Learning for Robotic Motion
Control. arXiv:1511.03791 (2015).

[51] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh,and Patrick McDaniel. 2018. Ensemble Adversarial Training: Attacks and Defenses. In ICLR.

[59] M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid. 2018. DeepRoad: GANbased Metamorphic Autonomous Driving System Testing. ArXiv e-prints (Feb.
2018). arXiv:cs.SE/1802.02295[52] Mark Utting and Bruno Legeard. 2007. Practical Model-Based Testing: A Tools
Approach. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

[60] Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating Natural Adversarial Examples. In ICLR.

[53] Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. 2018. Feature-Guided
Black-Box Safety Testing of Deep Neural Networks. International Conferenceon Tools and Algorithms for the Construction and Analysis of Systems (TACAS)(2018).

[61] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. 2018. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv preprint arXiv:1806.02891(2018).
[62] Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test
Coverage and Adequacy. ACM Computing Survey 29, 4 (Dec. 1997), 366–427.

131