DeepCT: Tomographic Combinatorial Testing for
Deep Learning Systems
Lei Ma1 , Felix Juefei-Xu2 , Minhui Xue3 , Bo Li4 , Li Li5 , Yang Liu6 , Jianjun Zhao7
Harbin Institute of Technology, China 2 Carnegie Mellon University, USA 3 Macquarie University, Australia
4
University of Illinois at Urbana–Champaign, USA 5 Monash University, Australia
6
Nanyang Technological University, Singapore 7 Kyushu University, Japan

I. I NTRODUCTION
The Deep learning (DL) system has achieved tremendous
progress over the past decade and become the state-of-the-art
technique for many cutting-edge intelligent applications. However, recent studies reveal that the robustness of DL systems
is a big concern. A DL system that obtains a high prediction
accuracy could still be vulnerable against adversarial attacks
with minor perturbations on inputs [1]. Given that more andmore safety- and security-sensitive applications start to adopt
DL, deploying DL without thorough testing can potentiallylead to severe consequences. Although it is highly desirable
to systematically verify and provide formal guarantees on thesafety and robustness of a DL system, the current attempt
shows that the verification of DL systems [2] is still at an
early stage and could be exceptionally challenging due to thehuge runtime state space.
Software testing is widely adopted in traditional software
industry due to its scalability, effectiveness, and usefulness
for defect and robustness issue detection. Software testing
has recently started to be applied for defect detection ofdeep neural networks (DNNs) [3]–[8]. Combinatorial testing(CT) is an effective testing technique that systematically
explores the input configuration combinations, which can welltrade off the defect detection capability and huge testing978-1-7281-0591-8/19/$31.00 c 2019 IEEE

f1

DeepCT

DeepCT

DeepCT

g1out
g2out

Pooling

f2
Pooling

Abstract—Deep learning (DL) has achieved remarkableprogress over the past decade and has been widely applied tomany industry domains. However, the robustness of DL systems
recently becomes great concerns, where minor perturbation on
the input might cause the DL malfunction. These robustness
issues could potentially result in severe consequences when a DL
system is deployed to safety-critical applications and hinder the
real-world deployment of DL systems. Testing techniques enable
the robustness evaluation and vulnerable issue detection of a DL
system at an early stage. The main challenge of testing a DL
system attributes to the high dimensionality of its inputs andlarge internal latent feature space, which makes testing eachstate almost impossible. For traditional software, combinatorial
testing (CT) is an effective testing technique to balance the testing
exploration effort and defect detection capabilities. In this paper,
we perform an exploratory study of CT on DL systems. We
propose a set of combinatorial testing criteria specialized for
DL systems, as well as a CT coverage guided test generation
technique. Our evaluation demonstrates that CT provides a
promising avenue for testing DL systems.
Index Terms—Deep learning, combinatorial testing, robustness

Pooling

1

gqout

fp

Fig. 1: For CNNs, the features are composed of multiple convolutional and pooling layers. DeepCT examines the interactions amongneurons within each layer, just like a computed tomography scan.

space searching effort. CT has been successfully applied to
traditional configurable software systems [9]. To minimize
the test suite size while obtaining desired CT coverage, CT
often transforms the test generation into a constraint solving
problem. The ability to handle constraints becomes crucial for
real-world applications since most of the real-world systems
are subjected to constraints involving multiple parameters andconfigurations. Hence, research on CT [10] has experienced
a shift to the constrained CT which breaks down as metaheuristic approaches [11], SAT-based approaches [12], andgreedy approaches [13]. In this paper, we take the first step
towards exploring whether CT is helpful for testing DL
systems. We propose a set of CT coverage criteria for DL
systems, as well as a CT coverage guided test generation
technique. Our initial evaluation results demonstrate that CT
is indeed a promising direction for testing DL systems.
II. BACKGROUND AND M OTIVATION
To better motivate this work, we discuss the intentions of
this paper from the following two perspectives:• Per 1 : Why ‘tomographic’, i.e., CT focuses on more
intensive testing within each layer?
• Per 2 : Why ‘combinatorial’, i.e., CT systematically examines the interactions among neurons within each layer?
For Per 1 , we rely on some observations from the viewpointof DNN formulation and properties, which allow us to arrive
at a testing method that is tailored towards being tomographic.
Let us take a convolutional neural network (CNN) for image
processing as an example,1 and the discussion followed can beeasily generalized to more general deep learning systems composed of feed-forward DNNs and recurrent neural networks.
Figure 1 shows an abstraction of a particular CNN model with
1 CNN is commonly used for image processing, which is a special case of feed-forward
deep neural network, with local receptive field and weight sharing.

535

SANER 2019, Hangzhou, Chinamultiple layers connecting input features and output features.
Let fl be the l-th image feature with dim(fl ) = n × 1 and
(k)
(k)
(k)
gl be the l-th feature map with dim(gl ) = nl × 1. The
convolutional layer can be mathematically represented by:
!!!
qk−1
qk−2
X (k)
X (k−1)(k)
gl = ξ
Wl,l0 ? ξ
Wl,l0 ? ξ · · · fl0
,
l0 =1

l0 =1

(a)

where ξ is the activation function (e.g., ReLU, followed by(k)(k−1) 0a pooling operator, such as gl (x) = kgl(x ) : x0 ∈N (x)kp , p = 1, 2, or ∞).
For CNNs, a major assumption is that the data (image,
videos, audio) are compositional with patterns that are: local,
stationary, and most importantly, multi-scale (hierarchical).
CNNs leverage the compositionality structure of the data andextract compositional features and feed them to the subsequentclassifier, recommender, etc., in an end-to-end fashion. The
design of CNNs (and DNNs in general) is entirely driven
by these ‘rules’ found in the data patterns. Let us take themulti-scale property as an example, simple structures are
combined to compose slightly more abstract structures, andso on, in a hierarchical and layer-by-layer way. Inspired bybrain visual primary cortex [14], features learned by CNNs
become increasingly more complex at deeper layers [15]. Thisis especially interesting because weights in various layers ofa CNN are structured, hierarchical, and play different rolesin the entire DL system. At any hidden layer, the neurons
will consolidate all the previous information and pass along
to the subsequent layers as input data with higher levels of
abstraction. This is also confirmed by visualizing what theneurons learn at each layer [15], from simple combinationsof corners and edges, to rich textures and 3D structure, tohigh-level structures like object parts.
Therefore, to test the robustness of the DNN model, it could
be a good practice to test it layer by layer and systematically
explore its latent feature space just like a computed tomography scan, since different layers of the DNN function at
different scales. Hence, ‘tomographic’.
As for Per 2 , there are several interesting aspects to discuss.
First, if we zoom into two adjacent layers in a DNN, all theneurons only interact (in the forward pass) with neurons fromthe next layer, and during back-propagation, all the neurons
only interact (in the backward pass) with neurons from theprevious layer. We can treat this local two-layer structure as
an undirected bipartite graphical model, where no interactions
happen within that layer. This is analogous to what ‘restricted’refers to in the restricted Boltzmann machine. Figure 2 showsinteractions between two adjacent layers using standard convolution, two variants (locally connected convolution and tiled
convolution), as well as the fully connected version which isessentially the DNN case. It is obvious that there are no neuralinteractions within each layer.
Previous DL testing methods [3], [16] mostly focus on
the measurement of the single-neuron behavior of the entire
network at a time, and the coverage of a particular neuron
illustrates one dimension of the entire landscape, without monitoring the interactions between neurons. In DNNs, although

(b)

(c)

(d)

Fig. 2: Interactions between two adjacent layers for (a) standard
convolution, (b) locally connected convolution, (c) tiled convolution,and finally (d) fully connected case (DNN).

there are no direct (tangible) interactions between the logic
units (neurons) within each computational layer, there could
be logical (intangible) interactions between the logic units,
where neurons of current layer altogether decide the logic ofneurons of its next layer. We want to capture and examine
these intangible interactions among neurons of each layerusing CT. Hence, ‘combinatorial’.
Second, covering the output space of a layer of neurons (i.e.,
the testing input feature space of its next layer) in DL is challenging because (1) the number of output values of a neuron
is often huge (e.g., infinite for continuous case), (2) neuron
combination exponentially grows as neuron size increases,
causing exhaustive enumeration infeasible. To sidestep, we
discretize the space of the output values and only observe
whether a neuron is activated. The effective testing of neurons’
combinatorial behavior instead of an exhaustive strategy iswhere CT comes into play. We adopt CT to systematicallytest diverse neuron interactions within each layer to uncoverdefects while reducing the number of test inputs that have to be
executed. Due to the aforementioned discussions, we propose
to perform tomographic CT to slice up the DL system andexamine layer-wise neural behavior as depicted by Figure 1.
The contributions of the paper are summarized as follows:• We propose a set of combinatorial testing criteria specialized for DL systems.
• We also propose a general CT coverage guided test
generation technique for DL systems.
• We perform studies on DeepCT to demonstrate the potential usefulness of CT for DL testing.
To the best of our knowledge, this is the first work to explore
the feasibility and usefulness of CT for DL systems.
III. C OMBINATORIAL T ESTING C RITERIA
We use N = {n1 , n2 , . . .} and T = {x1 , x2 , . . .} torepresent the set of neurons and test inputs of a DL system,
respectively. Let φ(n, x) be a function that returns the output
value of a neuron n ∈ N given a test input x ∈ T . For a DNN
with m layers, we use Li to denote the set of all neurons on
its i-th layer li (1 ≤ i ≤ m). Our CT coverage criteria for
DNNs are motivated by the combinatorial coverage metrics intraditional software testing, and the ways that a DNN processes
an input through transformations layer by layer. Similar toprevious work [3], [16], we consider the neuron n to be
activated (resp. deactivated) given x if φ(n, x) ∈ (0, ∞) (resp.
φ(n, x) ∈ (−∞, 0]). We use A(ni , x) ∈ {0, 1} to represent the536

activation status of a neuron ni given x, where 1 corresponds
to activation and 0 deactivation. To test the functionality of alayer, we first introduce the neuron-activation configuration.

Given a test set T and the neurons Li of the i-th layer, we
use Θ(t, Li ) to denote the set of all t-way combinations ofneurons in Li . Each element θ (i.e., θ ∈ Θ(t, Li )) is a set oft neurons, with a total of 2t neuron-activation configurations.
Furthermore, We use Θfull (t, Li , T ) (i.e., Θfull (t, Li , T ) ⊆Θ(t, Li )) to represent those t-way combinations of neurons ofLi , where all the neuron-activation configurations of θfull ∈Θfull (t, Li , T ) are fully covered by T . For a t-way neuron
combination θ ∈ Θ(t, Li ), we use C (t, θ, T ) to represent
the set of neuron-activation configurations of θ covered by
T . Next, we define the t-way combinatorial testing criteria.
Definition III.2 (t-way combination sparse coverage). We define the t-way combination sparse coverage as the percentageof t-way neuron combinations in Li , of which all the neuronactivation configurations are covered by T ,SparseCov(t, Li , T ) =|{θ ∈ Θ(t, Li )|θ ∈ Θfull (t, Li , T )}||Θ(t, Li )|Example III.1. Let Li = {n1 , n2 , n3 , n4 } be a set ofneurons in the same layer. Each row j in Figure 3 corresponds to the neuron activation status given a test of T ,denoted by A(Li , xj ). There are in total six 2-way combinations of neurons in Li , {n1 , n2 }, {n1 , n3 }, {n1 , n4 },{n2 , n3 }, {n2 , n4 } and {n3 , n4 }. Each 2-way combination has
four neuron-activation configurations (0, 0), (0, 1), (1, 0) and(1, 1). Among the six 2-way combinations, only the full neuronactivation configurations of {n1 , n2 }, {n1 , n4 }, {n2 , n3 } and{n3 , n4 } are covered by T . Therefore, the 2-way combination
sparse coverage for T on Li is 66.6%.
Since the t-way combination sparse coverage cannot takethe coverage within each combination of neurons into account,next we introduce the t-way combination dense coverage.
Definition III.3 (t-way combination dense coverage). Fora set of test inputs T and a set of neurons Li , the t-way
combination dense coverage is defined as below,Pθ∈Θ(t,Li ) |C(t, θ, T )|DenseCov(t, Li , T ) =2t |Θ(t, Li )|Example III.2. Consider the neuron activation status of Li
given test set T in Figure 3. Since there are six 2-way combinations of neurons in Li and each has four neuron-activation
2 Note that k is not necessarily the number of neurons of the layer L . In the case ofit-way combination of Li , k equals to t.

Li-1

Li

Lm

n1

...

n2 ...

Test Inputs

n3T

Input Layer

n1
0
0
1
1

n2
0
1
0
1

n3
0
0
1
1

n4
0
1
0
1

...

...

n4

...

Definition III.1 (Neuron-activation configuration). For a setof neurons M = {n1 , n2 , . . . , nk } ⊆ Li of the i-th layer
li , a neuron-activation configuration of M is a tuple c =(b1 , b2 , ..., bk ), where bi = {0, 1}.2 A neuron-activation configuration c = (b1 , b2 , ..., bk ) is covered by T if there exists
at least a test input x ∈ T , such that bi = A(ni , x) for all i,where 1 ≤ i ≤ k.

L1

Hidden Layers Output Layer

Fig. 3: Example of the neuron activation status of Li in layer li (thatcontains 4 neurons) given test inputs T .

configurations, there are 24 neuron-activation configurations
in total. The test set T can cover 20 configurations andthe neuron-activation configurations that are not covered are{n1 , n3 } = (0, 1), {n1 , n3 } = (1, 0), {n2 , n4 } = (0, 1) and{n2 , n4 } = (1, 0). Therefore, the 2-way combination dense
coverage for T is 83.3%. Recall that the 2-way combination
sparse coverage for T is 66.6%.
Definition III.4 ((p, t)-completeness). For a set of neurons Li
and test inputs T , (p, t)-completeness is defined as the proportion of those θ ∈ Θ(t, Li ), where the covered t-way
neuron activation configuration ratio of θ is at least p (i.e.,C(t, θ, T )/2t ≥ p).
Example III.3. We again take Figure 3 as an example. For
2-way combinations of neurons, the covered 2-way configuration ratio for {n1 , n2 }, {n1 , n4 }, {n2 , n3 } and {n3 , n4 } are
100%, and 50% for {n1 , n3 } and {n2 , n4 }. According to theDefinition III.4, the (0.5, 2)-completeness of Li is 100% andthe (1, 2)-completeness for Li is 66.6%.
Although the combinatorial testing criteria defined above
consider neurons for a layer, they could be easily generalized
to the whole neural network level layer by layer. We leveragethese criteria to evaluate the testing effectiveness from theperspective of CT and to further guide test generation.
IV. CT ROBUSTNESS T ESTING OF D EEP L EARNING
Our proposed CT coverage criteria are general for both test
suite evaluation and test generation guidance. In this study,
we consider a typical problem of using CT criteria to test
the adversarial robustness of DNNs designed for the generalpurpose image classification [17]. Given an input x that canbe correctly classified by a DNN, the adversarial robustness
property is concerned with whether there exists another input
x0 close to x, with respect to some distance metrics (e.g.,
L∞ -norm), where x and x0 are classified to different classes by
the DNN. Such an input x0 , once exists, is called an adversarial
example of x and the DNN is not locally robust at x.
Let C(x) denote the class to which x is classified by DNNs.
Formally, a DNN is d-locally-robust at an input x w.r.t. a
distance parameter d iff we have the following [17]3 :∀x0 : kx0 − xk ≤ d ⇒ C(x) = C(x0 )3 Adversarial robustness could be analyzed by checking the local robustness property.
Therefore, the rest of this paper focuses on testing the local robustness of DNNs.

537

LP constraint solving based test generator, which we use toinvestigate whether CT and our proposed criteria are usefulfor testing DNNs.
In particular, we mainly investigate whether DeepCT and
our criteria are useful for adversarial example detection
by local robustness analysis. We use the publicly available
dataset MNIST and two pre-trained DNN models [4] that
achieve competitive prediction accuracy. The two studied
DNNs contain 3 (64*32*64 with 55,082 parameters) and5 (84*42*64*42*84 with 79,454) fully-connected hidden layers, and obtain 99.965%, 99.872% training accuracy, and97.63%, 97.51% test accuracy respectively. For the DNNs’
local robustness analysis, we randomly seed 1,000 tests fromMNIST accompanied test set as the study subject, which canbe correctly handled by our studied DNNs.

Algorithm 1: DeepCT Test Generation
INPUT: DNN N , t-way, CT Criteria CTC , seeding Tests Ts
OUTPUT: Passed Test Suite T , Adversarial Test Suite T 00← {}table←initialize CT coverage table(Ts , t-way, CTC )3: for t ∈ Ts do
4:
Twork ← {}5:for each layer neuron set Li in N do
6:
CT table ← update CT coverage(CT table, Twork , CTC )7:
CT targets←calculate CT targets(CT table, Twork , Li , CTC )8:while CT targets 6= ∅ ∧ time budget exists do
9:
ctj ←random select(CT targets)10:
gen tests←TestGen(t, ctj )11:covered targets ← cal cov targets(CT table, Twork , gen tests, CTC )12:
CT targets←CT targets - covered targs
13:
update TestSuite(gen tests, Twork , T , T 0 )0
14: return T, T
1: T ← {}, T
2: CT

One approach to analyze local robustness is through random
testing. However, random testing is often ineffective even if
a large number of tests are generated [4]. To systematicallygenerate tests to detect local robustness issues of a given input,
we propose a combinatorial testing technique for test suite
generation of a DL system. Algorithm 1 shows the details of
our CT coverage guided test generation. Given a seeded test
set, t-way, and CT criterion as input, the CT coverage tableof the whole DNN is first initialized (see Line 1-2). Then,the test generation iteration starts for each seeding test guided
by the CT coverage layer by layer. For each candidate layer,
the coverage is analyzed on the generated tests so far. The
coverage table is updated and the uncovered CT targets of alayer li are calculated (Line 6-7). After randomly selecting atarget ctj , we try to generate tests to cover ctj and analyze
all the coverage targets they reach. Note that a test casemight cover multiple CT targets in t-way testing, and thegenerated tests might not cover any CT targets (Line 9-12).
Before attempting on the next CT target, we check whether
the generated tests contain adversarial examples and update thegenerated test suite for T , T 0 , and the test working set Twork
accordingly (Line 13). The test generation iteration continues
until CT coverage targets are covered or the time limit hits.
Our test generation technique to cover the specific CT
coverage target is general without assumption on specific types
of DNN structures (e.g., activation functions) or test generation
methods (e.g., search-based testing [18], guided random testing [19], symbolic constraint solving based testing [20]). To
demonstrate the CT coverage guided test generation is helpful
for detecting adversarial examples, this study assumes that
DNN uses the popular ReLU activation function and adopts theconstraint solving based (i.e., the Cplex solver) test generator[4]. In particular, a CT coverage target is encoded as thelinear constraints with the object to minimize the L∞ -norm
perturbation distance on a seeding input.
V. E VALUATION
We have implemented DeepCT, a DL tomographic combinatorial testing framework that performs automated test
generation for DNNs based on Keras (ver.2.1.3) and Tensorflow (ver.1.5.0). The current version of DeepCT provides aA. Random Testing
Although previous work [4] advocated that random testing (RT) is ineffective in detecting the adversarial robustness
issues of DNNs, we believe that RT is easy to use and scalable,
which is worth a first shot before further in-depth analysis.
Therefore, our first step performs random test generation to
analyze the robustness of two studied DNNs (i.e., DNN 1 and
DNN 2 ) on the 1,000 seeded tests. To be specific, we randomly
generate 10,000 tests for each seeded test and analyze whether
robustness issues could be detected.4 The experiment results
show that the RT is already able to detect robustness issues
on 194 seeded tests on DNN 1 and 178 on DNN 2 with a totalof 266 unique issues, 106 of which are shared issues on both
DNN 1 and DNN 2 . This is consistent with our intuition thatthe robustness of a DNN on handling different test input could
be different. A test input near a DNN’s decision boundary
could trigger robustness issues more easily. Our experimental
results confirm this observation and indicate that RT could be
useful to detect the fragile test input as the first attempt.
B. DeepCTFor the 1,000 seeded tests, we filter out those 266 tests
whose robustness issues can already be detected by random
testing. For the remaining 734 tests, we randomly sample 50
tests for further analysis on d-locally-robustness by DeepCT(where d = 0.15, and we use 2-way defined CT criteria,also see Algorithm 1) in line with the corresponding tests
generated by random testing. DeepCT incrementally generates
tests to cover CT coverage targets layer by layer. Table I
summarizes the averaged results. Columns 4-7 show theobtained coverage of 2-way sparse, 2-way dense, (0.5,2)completeness, and (0.75,2)-completeness coverage of DeepCT
under the corresponding criterion setting. Column 8 gives theaccumulated total number of generated tests and Column 9shows the corresponding detected adversarial tests ratio.
Overall, for both studied DNNs, random testing achieves
fairly low coverage of all the evaluated criteria and DeepCT
achieves much higher coverage although with a similar number
4 Each of 784 pixels of the test image is normalized to range [0, 1]. In this study, weallow the random perturbation of each pixel within range [−0.15, 0.15].

538

TABLE I: The obtained CT coverage and detected adversarial issues
by random testing and DeepCT. The coverage is calculated byconsidering the 2-way CT targets of all hidden layers of the DNN
instead of a single layer.

DNN 1

DNN 2

Testing 2-way Combinatorial Testing Coverage (%)
Method SparseCov DenseCov (0.5,2) (0.75,2)Random
2.28
34.95
33.75
3.75
CT l1
60.27
81.56
95.01
70.98
CT l2
76.94
91.98
99.67
91.30
CT l3
93.62
98.23
100.0099.32
Random
1.18
32.56
26.98
2.10
CT l1
46.96
75.10
91.95
61.50
CT l2
68.91
87.52
98.64
82.55
CT l3
97.15
99.05
100.0
99.03
CT l4
97.41
99.11
100.0
99.03
CT l5
97.81
99.21
100.0
99.03

Accu.
Adv.
Tests Per.(%)
10,000 0.00
4,073
0.29
6,768
2.17
8,032
9.91
10,000 0.00
8,547
1.87
11,573 3.53
13,129 8.84
13,217 9.35
13,351 9.98

a promising avenue for testing DL systems. Our future work
would perform more in-depth investigation on the effect of t in
t-way CT for adversarial example detection, and incorporate
more scalable CT test generators for real-world DL systems.
ACKNOWLEDGEMENTS
This work was partially supported by 973 Program (No.
2015CB352203), Fundamental Research Funds for the Central
Universities (No. AUGA5710000816) of China, and JSPS
KAKENHI Grant 18H04097. We gratefully acknowledge thesupport of NVIDIA AI Tech Center (NVAITC) to our research.
R EFERENCES

of tests. In addition, DeepCT increases the coverage as morelayers of a DNN is tested with more tests generated. We
could also see that, DeepCT could detect 0.15-local-robustness
issues on all of the 50 studied tests for both DNNs with areasonable number of tests. To be specific, Table I showsthat random testing achieves only 2.28% and 1.18% 2-way
sparse coverage on two studied DNNs. Compared with (0.5,2)completeness coverage, (0.75,2)-completeness coverage is alsomuch lower, indicating that random testing does not deeply
cover many of the neuron activation configurations of 2-way
neuron combinations. In comparison, DeepCT obtains 60.27%
and 46.96% 2-way sparse coverage even only the first hidden
layer is analyzed, with a reasonable test suite size whilealready being able to detect some adversarial examples forall seeded tests. As described in Algorithm 1, when DeepCT
analyzes the second hidden layer l2 , it first analyzes thecoverage obtained by tests generated by all previous layers,and generates tests to cover the uncovered target on layer l2 . In
the two studied DNNs, after generating tests for layer l2 , 2-way
sparse coverage of the whole network increases by 16.67%
and 21.95%, respectively. Similar coverage improvement could
also be observed by other coverage criteria. For example, theobtained (0.75,2) coverage indicates that more than 80% the 2way neuron interactions are mostly covered deeply. When thefirst three layers of the two studied DNNs are analyzed, the 2way sparse coverage reaches 93.62% and 97.15%, respectively,
with many of the adversarial examples detected. For DNN 2 ,only about 220 new tests in total are created when analyzing
layers l4 and l5 . As for the detected adversarial examples,adversarial ratio gaps of l2 and l3 are much larger than otherlayers for both DNNs. This indicates that the different layers
might contribute differently to detect adversarial examples by
CT, and some layers might need to be more intensively tested.
VI. C ONCLUSION
Combinatorial testing is a well-established and useful technique in traditional software testing. Rather than exhaustively
searching all the combinations of input space, CT focuses on
testing the interactions of inputs, aiming to reduce the test
suite size while obtaining satisfiable defect detection abilities.
This paper initiates the first study to explore the usefulness of
CT for testing DL systems. Our results show that CT provides[1] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing
adversarial examples,” in ICLR, 2015.
[2] W. Xiang, P. Musau, A. A. Wild, D. Manzanas Lopez, N. Hamilton,
X. Yang, J. Rosenfeld, and T. T. Johnson, “Verification for Machine
Learning, Autonomy, and Neural Networks Survey,” ArXiv e-prints, p.
arXiv:1810.01989, Oct. 2018.
[3] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox
testing of deep learning systems,” in Proceedings of the 26th Symposium
on Operating Systems Principles, 2017, pp. 1–18.
[4] Y. Sun, X. Huang, and D. Kroening, “Testing Deep Neural Networks,”
ArXiv e-prints, Mar. 2018.
[5] L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su,
L. Li, Y. Liu, J. Zhao, and Y. Wang, “Deepgauge: Multi-granularity
testing criteria for deep learning systems,” in The 33rd IEEE/ACM
International Conference on Auto. Software Engg. (ASE), 2018.
[6] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li,
Y. Liu, J. Zhao, and Y. Wang, “DeepMutation: Mutation Testing of
Deep Learning Systems,” in The 29th IEEE International Symposium
on Software Reliability Engineering (ISSRE), 2018.
[7] X. Xie, L. Ma, F. Juefei-Xu, H. Chen, M. Xue, B. Li, Y. Liu, J. Zhao,
J. Yin, and S. See, “DeepHunter: Hunting Deep Neural Network Defects
via Coverage-Guided Fuzzing,” arXiv preprint arXiv:1809.01266, 2018.
[8] X. Du, X. Xie, Y. Li, L. Ma, J. Zhao, and Y. Liu, “Deepcruiser:
Automated guided testing for stateful deep learning systems,” arXiv
preprint arXiv:1812.05339, 2018.
[9] C. Nie and H. Leung, “A survey of combinatorial testing,” ACM
Computing Surveys (CSUR), vol. 43, no. 2, p. 11, 2011.
[10] B. S. Ahmed, L. M. Gambardella, and et al., “Handling constraints
in combinatorial interaction testing in the presence of multi objective
particle swarm and multithreading,” IST, vol. 86, pp. 20–36, 2017.
[11] Y. Jia, M. B. Cohen, M. Harman, and J. Petke, “Learning combinatorialinteraction test generation strategies using hyperheuristic search,” inIEEE/ACM 37th Intl. Conf. on Softw. Egg. (ICSE), 2015, pp. 540–550.
[12] A. Yamada, T. Kitamura, C. Artho, E.-H. Choi, Y. Oiwa, and A. Biere,“Optimization of combinatorial testing by incremental sat solving,” inThe 8th Intl. Conf. on Softw. Test. Verif. & Valid. (ICST), 2015, pp. 1–10.
[13] J. Czerwonka, “Pairwise testing in real world,” in 24th Pacific Northwest
Software Quality Conference, vol. 200. Citeseer, 2006.
[14] D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction
and functional architecture in the cat’s visual cortex,” The Journal ofphysiology, vol. 160, no. 1, pp. 106–154, 1962.
[15] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and
A. Mordvintsev, “The building blocks of interpretability,” Distill, 2018.
[16] Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testingof deep-neural-network-driven autonomous cars,” in Proc. of the 40thInternational Conference on Software Engineering, 2018, pp. 303–314.
[17] G. Katz, C. W. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer,“Reluplex: An efficient SMT solver for verifying deep neural networks,”
CoRR, vol. abs/1702.01135, 2017.
[18] P. McMinn, “Search-based software testing: Past, present and future,”in ICST’11, 2011, pp. 153–163.
[19] L. Ma, C. Artho, C. Zhang, H. Sato, J. Gmeiner, and R. Ramler, “Grt:
Program-analysis-guided random testing (t),” in The 30th IEEE/ACM
Intl. Conf. on Auto. Softw. Engineering (ASE), 2015, pp. 212–223.
[20] R. Baldoni, E. Coppa, D. C. D’Elia, C. Demetrescu, and I. Finocchi, “A
survey of symbolic execution techniques,” ACM Comput. Surv., vol. 51,
no. 3, 2018.

539